Performance Tuning FME

Introduction

This article series covers the many ways of performance-tuning FME, in different FME applications, and at different stages of project development.
On a computer, work is carried out using system resources. If performance is the time taken to carry out useful work, then excessive use of resources, or time lost on useless work, hinders performance.
Many such factors are diagnosed or reported via the FME log file, so this series covers how to read a log file in some detail. This particularly helps to find out whether there is an overall performance slow down, or a bottleneck at one stage of the process.
 
Note that previously this article series was contained within a single article. We have since broken the article into separate articles so we could expand on each topic as well as it would be easier to find via search. Click the links below to jump to the topic of interest. 
 

Performance Tuning Articles

System Hardware and Configuration

The hardware upon which FME is run, and its configuration, can greatly affect performance.
 

Workspace Authoring & Design Patterns

There are a few tools available in FME to help a workspace run faster, while it is being authored and debugged, thus speeding up the authoring process. As well, it is important to design your workspace for maximum performance as you create it. There are various patterns of design that promote better performance.
 

Post-Mortem Performance Tuning

Analyzing the results of a process in order to assess or improve future performance is called post-mortem debugging. In FME, the key result of a process is the log file.
Therefore, before being able to tune a workspace, it's vital to understand how to read an FME log file. Without this knowledge, a user will often jump to incorrect conclusions about a translation and start looking for performance issues in the wrong places.
 

Database Performance Tuning

Databases are an important component of many datasets, and the log file will help us determine how good our database performance is, plus how well FME is interacting with the database.
 

FME Updates and Improvements

As FME is developed and refined over time, it gains a number of performance improvements. These can be classified as two different types of upgrades, automatic and premeditated.
 

FME Memory Management

Memory Management is mostly an automatic process that the FME engine carries out during a translation. However, there are log file messages that relate to memory management and can be useful to inspect.
 

Additional Performance Tuning Tips

In FME Workbench, you can run a profile of your FME Workspace to obtain a detailed log of how much time is spent in each underlying factory or function. Use Tools -> Edit Header to add the directive: FME_PROFILE_RESULT_CSV i.e:

FME_PROFILE_RESULT_CSV C:\TEMP\profile.csv

Notes:

• The profile directive is not supported on FME Flow
• Don't forget to remove the profile directive for your production workspace since profiling also has an impact on performance!

It's sometimes worth using a Performance Monitoring tool (such as PerfMon) to log the CPU and memory usage of a process.

Additional Resources

Use Data Integration Best Practices [FME Academy Trail]
How FME Improves Performance with Bulk Mode [Documentation]
Feature Caching and Performance [Article]