Snowflake Writer: "java.lang.OutOfMemoryError"

SteveatSafe
SteveatSafe
  • Updated

FME Version

  • FME 2020.2

Symptom

When attempting to write data to Snowflake an error is encountered and the translation fails with the following error message:
Snowflake Spatial Writer: Committing feature 32844
java.lang.OutOfMemoryError: Java heap space

 

Cause

The java.lang.OutOfMemoryError has to do with the number of features being passed to the Snowflake writer in one transaction. It will most likely occur with large spatial data representing large complex polygons due to the amount of memory that is needed to store the geometries in memory before passing it to the database for writing/committing. By default, the Snowflake writer will send 100,000 features in one transaction before it performs a COMMIT.    


Solution

Decrease the Writer Parameter Features Per Transaction to a smaller number via the writer parameters in the Navigator.  

If you are writing less than 100,000 features, for example, 40,000 features, and you are seeing the error message, try reducing the Features Per Transaction to 20,000.  Again, you may need to reduce further, say to 10,000 features, if the error persists. 

If you are writing more than 100,000 features, for example, 500,000 features, try reducing the Features Per Transaction to 50,000 and test. If it passes you may want to increase this value to get better throughput/performance. If it fails again, lower the Features Per Transaction to 25,000 and test. Repeat if necessary.

Additionally, the FME_JVM_MAX_HEAP_SIZE environment variable can be modified to specify the amount of memory allocated. See the Usage Notes section of the Snowflake Spatial Reader/Writer documentation for more information. 

If this solution does not resolve the problem, please contact Safe Software Support

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.