Considerations
When writing data to a Snowflake database, users may encounter the error: java.lang.OutOfMemoryError: Java heap space
This typically occurs during translation when a large number of features are written in a single transaction.
Why This Happens
The Snowflake writer uses Java to handle transactions. By default, it attempts to commit 100,000 features at a time. If the spatial data includes large or complex geometries (such as detailed polygons), memory usage increases significantly. This can exceed the default Java heap space allocation, resulting in a memory error.
This behavior is more common when:
Writing complex geometries like polygons or multipolygons
Working with large volumes of spatial data
Using the default Features Per Transaction setting of 100,000
How to Resolve It
1. Lower the Features Per Transaction
Reduce the number of features processed in each transaction:
If writing fewer than 100,000 features (e.g., 40,000), try setting Features Per Transaction to 20,000 or 10,000.
If writing more than 100,000 features (e.g., 500,000), reduce the setting to 50,000. If that fails, lower it further to 25,000 or less.
To change this setting:
In FME Workbench, open the Navigator
Locate the Snowflake writer
Modify Features Per Transaction
Adjust and re-run until the translation completes successfully.
2. Increase Java Heap Size
Set the FME_JVM_MAX_HEAP_SIZE environment variable to allocate more memory to Java. This can allow larger transactions without hitting memory limits.
See the Snowflake Reader/Writer Usage Notes for details.
Additional Notes
Reducing the transaction size may slightly affect performance, but it helps ensure successful translation with large or complex spatial data. You can experiment to find the best balance between performance and stability for your dataset.
If problems continue after adjusting these settings, contact Safe Software Support.