Performance Tuning FME: FME Memory Management

Introduction

Memory Management is mostly an automatic process that the FME engine carries out during a translation. However, there are log file messages that relate to memory management and can be useful to inspect. For other performance tuning tips & tricks, see Performance Tuning FME.

FME Memory Configuration

At the start of the translation, FME determines the amount of memory available and plans how to use that memory to give optimum performance. Two lines in the log file indicate these plans:

INFORM|FME Configuration: Start freeing memory when the process exceeds 71.99 GB
INFORM|FME Configuration: Stop freeing memory when the process is below 53.99

Note that this refers to virtual memory (the system's mix of physical memory and disk space). In the above example, FME plans to use as many resources as required, up to a limit of 71.99 GB. At that point, it will swap, moving data out of memory and onto the temporary disk folder.
When it has released enough memory to go below 53.99GB of usage, it will again start using memory instead of caching to disk.
Because caching to disk is expensive, it's best for performance if FME can stay within the maximum limits. If it cannot do so, then it will record its actions in the log.

Optimizing Memory Usage

If you see either of these messages in the FME log:

INFORM|ResourceManager: Optimizing Memory Usage. Please wait...
INFORM|Failed to free sufficient memory to reach the process usage limit

...they are evidence that FME has to optimize its memory use. The first message reports that FME has reached its upper limit and is now releasing memory by caching to disk.
The second message warns that FME has not been able to release enough memory to reach the lower limit. This may be because extra applications are starting to take up the memory that FME is expected to use. If your FME task is completed in the expected time frame with the above error messages, real problems can occur for a number of reasons:

• Insufficient resources:
  • Insufficient Specification: Your computer may simply lack the resources (memory and disk space) required to process the data.
  • Competing Applications: Check the task manager to see if other applications are taking up resources, or you might also have other FME jobs running (particularly if you are seeing this on FME Flow (formerly FME Server)). Where possible, don't start new applications once FME is running. They may take up resources that FME was expecting to be able to use.
  • Memory Fragmentation: Fragmentation occurs over time as applications start and stop. If your system memory has become fragmented, it's harder for FME to allocate contiguous memory. You can identify this if your job runs fine after a system restart, but the same job starts to report memory optimization later in the day. Recourse: restart your computer.
• Workspace Issues:
  • Workspace Complexity: Multiple blocking transformers (Group By) can cause extreme memory use. If possible, you might need to increase your resources or redesign your workspace.
  • Debugging Options: Turn off feature caching or other debugging tools like Inspector or Logger transformers.

Note: Using the AttributeKeeper transformer to keep only attributes needed during the workflow or for the output drastically improves performance. 
If you consistently see the Optimizing Memory Usage for the same FME workspace, then share the workspace and data with us (Safe Support), and we can try and fix the problem.

FME Flow Engine Parameters 

The FME Process Monitor periodically restarts FME Flow Engines to control any growing memory concerns. Two parameters control the frequency of FME Flow Engine restarts to clear the memory they may build up:

• MAX_TRANSACTION_RESULT_SUCCESSES - maximum number of successful result transactions to accept from the service before shutting down or restarting the service.
• MAX_TRANSACTION_RESULT_FAILURES - maximum number of failed result transactions to accept from the service before shutting down or restarting the service.

e.g.
MAX_TRANSACTION_RESULT_SUCCESSES=100 will reset Engines after 100 successful jobs.
MAX_TRANSACTION_RESULT_FAILURES=10 will reset Engines after 10 failure jobs.

These settings are found in FMEFlow/Server/fmeFlowConfig.txt – note that FME Flow must be restarted for any changes to take effect.