Performance Tuning FME: FME Memory Management

Introduction

Memory Management is mostly an automatic process that the FME engine carries out during a translation. However, there are log file messages that relate to memory management and can be useful to inspect. For other performance tuning tips & tricks, see Performance Tuning FME.
  

FME Memory Configuration

At the start of the translation, FME determines the amount of memory available to it and plans how to use that memory to give optimum performance. Two lines in the log file indicate these plans:

INFORM|FME Configuration: Start freeing memory when the process exceeds 71.99 GB
INFORM|FME Configuration: Stop freeing memory when the process is below 53.99

Note that this refers to virtual memory (the system's own mix of physical memory and disk space). In the above example, FME plans to use as many resources as required, up to a limit of 71.99GB. At that point, it will carry out its own swapping, moving data out of memory and onto the temporary disk folder.
When it has released enough memory to go below 53.99GB of usage, it will again start to make use of memory instead of caching to disk.
Because caching to disk is an expensive operation, it's best for performance if FME can stay within the maximum limits. If it cannot do so, then it will record its actions in the log.
 

Optimizing Memory Usage

If you see either of these messages in the FME log:

INFORM|ResourceManager: Optimizing Memory Usage. Please wait...
INFORM|Failed to free sufficient memory to reach the process usage limit

...they are evidence that FME has to optimize its memory use. The first message reports that FME has reached its upper limit and is now releasing memory by caching to disk.
The second message warns that FME has not been able to release enough memory to reach the lower limit. This may be because of extra applications starting and taking up the memory FME expected to use. 
In some cases (FME 2019.0 & earlier), the messages are spurious, and FME is running fine. If your FME task is completed in the expected time frame, then this is likely the case.
However, real problems can occur for a number of reasons:

• Insufficient resources:
  • Insufficient Specification: Your computer may simply lack the resources (memory and disk space) required to handle the amount of data being processed.
  • Competing Applications: Check the task manager to see if there are other applications taking up resources, or you might also have other FME jobs running (particularly if you are seeing this on FME Server). Where possible, don't start new applications once FME is running. They make take up resources that FME was expecting to be able to use.
  • Memory Fragmentation: Fragmentation occurs over time as applications start and stop. If your system memory has become fragmented, it's harder for FME to allocate contiguous memory. You can identify this if your job runs fine after a system restart, but the same job starts to report memory optimization later in the day. Recourse: restart your computer.
• Workspace Issues:
  • Workspace Complexity: Multiple blocking transformers (Group By) can cause extreme memory use. You might need to increase your resources or redesign your workspace, if possible.
  • Debugging Options: Remember to turn off feature caching or other debugging tools like Inspector or Logger transformers.

 
Note: By using the AttributeKeeper transformer to only keep attributes that are needed during the workflow or for the output, performance is drastically improved. 
 
If you consistently see the Optimizing Memory Usage for the same FME workspace, then share the workspace and data with us (Safe Support), and we can try and fix the problem.
 

Memory Redline Parameter

WARNING: The value of this parameter should NOT be adjusted unless you have exceptional circumstances or have been directed by Safe Software. Changing this value can cause unexpected behavior and reduced performance. In most cases - particularly FME 2018 or newer - no benefit accrues from adjusting the memory redline parameter
The Resource Manager automatically determines the optimal total memory the FME Engine process should use. It also dynamically allocates this total memory optimally to the algorithms within FME requesting it.
The FME_ENGINE_MEMORY_REDLINE directive is a hint to the FME Engine on how aggressive it should be in consuming memory. It takes a value between 0 and 1 (0.5 is the default value). For more aggressive memory usage, a value above 0.5 should be used. For reduced memory usage, a value below 0.5 should be used.
The risk in being too aggressive is the process running out of memory or the machine "thrashing" (a state of constant memory errors). The risk in being too conservative is that the process may take disproportionately longer to complete.
To set this value in FME Workbench, select Tools > Edit Header from the menubar and enter:
FME_ENGINE_MEMORY_REDLINE 0.5
...as the very first line. For FME Server Engines, see How to Control FME Flow Engine Memory Usage.