Resource Planning Essentials for FME Flow Deployments

Introduction

FME Flow installations require tailored technical specifications based on your organization's workflow and data requirements. Due to this variability, Safe Software does not provide universal specifications. IT teams can use this article as a reference for planning your deployment. 

When deploying to virtual machines, use this article in tandem with 'Technical Specifications for FME Flow on Cloud-based Virtual Machines (VM)'. 

Before You Start

Before installation, review the basics of FME Flow architecture and requirements.

Documentation:

• Planning an FME Flow Installation

Articles:

• FME Platform Technical Specifications 
• Planning and Performing an FME Flow Installation
• Host System Sizing
• A Guide to Choosing your FME Server Deployment Architecture 
• FME Flow Best Practices Guide for IT Ops and Flow Admins 

Workflow Analysis

To understand your FME Flow needs, start by analyzing your largest dataset and most resource-intensive workflow. More resources are required to read, write, and transform large and complex datasets. Complex transformations, especially those that temporarily stash data in memory, also demand more resources.

If available, an FME workspace translation log can provide valuable information about a translation process.

Articles:

• Post-Mortem Performance Tuning
• Performance Tuning FME: FME Memory Management

Planning a Deployment

FME Flow consists of multiple components (Core, Engine, Database, Web Application) that can be installed together or separately. Carefully consider your deployment type when planning hardware, as each component utilizes system resources differently.

See the Guide to Choosing your FME Flow Deployment Architecture for a full overview of deployment options. 

Express

An FME Flow Express deployment contains all FME Flow components on a single machine. This is the easiest starting point for planning technical specifications.

FME Flow Engines are responsible for running jobs and processing. Thus, hardware recommendations primarily focus on the resources available to the Engines.

General hardware recommendations, based on an Express installation.

These are general guidelines that may need to be adjusted for your workflow requirements. 

Distributed

In distributed deployments, FME Flow components are installed on separate machines. This allows precise control over performance as hardware can be scaled per component, but also introduces a lot more variability into technical specifications. Therefore, the following sections will provide only very basic guidelines for each FME Flow component. 

FME Flow Engines

Within the FME Flow architecture, FME Flow Engines are significant consumers of system resources. A simple distributed FME Flow architecture may install Engines on a separate machine to maximize performance.

Engine types (Standard, CPU Usage, Remote) each differ in location and cost, but all share common hardware requirements. Job throughput can be increased by adding additional Engines, while job performance can be enhanced by optimizing resources and architecture.

See Planning for Scalability and Performance

Engines may be installed on the same machine as the rest of the FME Flow components, on a separate machine, or a combination of both. Optimal performance requires careful consideration of both the location (proximity) and resources available to the Engines. See FME Flow Administration: Customization and Monitoring for a complete overview of Engine optimization. 

CPU

FME Flow Engines are single-threaded processes that maximize use of one CPU core.

A basic rule is that every FME Engine should have a dedicated CPU core. However, larger datasets, complex processes, or long jobs require at least one core per Engine. Smaller datasets and short jobs can accommodate fewer cores per Engine. Specific workflow designs leveraging parallel processing (multi-threaded) require multiple cores per Engine

For example:

• 1 CPU with 1 core: 1 FME Engine consumes up to 100% of the total CPU.
• 1 CPU with 2 cores: 1 FME Engine consumes up to 50% of the total CPU.
• 1 CPU with 2 cores: 2 FME Engines consumes up to 100% of the total CPU.
• 2 CPUs, each with 2 cores: 2 FME Engines consumes up to 50% of the total CPU.

Memory

FME Engine physical memory (RAM) requirements are highly variable and are ultimately best informed by a review of your workflows and data.

The minimum recommendation (8GB for 2 Engines) can be quickly exceeded by workflow demands. Ensure memory sufficiently meets engine requirements to avoid slower translation times and potential failures.

Consider other applications and processes running on the same machine as FME Flow, as concurrent processes sharing resources with the FME Engines can degrade performance.

FME Flow Core, Web Application, Database

The FME Flow Core, Web Application, and Database have relatively fewer resource requirements compared to Engines. In a distributed environment, these components can be installed together (recommended) or separately.

The FME Flow Core and Web Application primarily run lightweight java processes. In some scenarios, the maximum memory allocated per process can be changed in the configuration files. The FME Flow database stores basic configuration and metadata properties for FME Flow, with repositories and resources located in the FME Flow System Share.

Typically, all three components can function smoothly with 2-4 GB (RAM) and one CPU core per component. More resources are beneficial, especially on busier systems.

FME Flow System Share

The FME Flow System Share location is set at installation and cannot be changed afterward. It is important to allocate enough disk space to accommodate your repositories  and resource files. 

A minimum of 10GB of disk space is recommended, but this depends on your data requirements.

FME Flow TEMP Folder

The FME Flow TEMP folder stores data that is not saved permanently (as in the FME System Share). A minimum of 10GB of disk space is recommended, but this depends on your data requirements. Frequent clean-up tasks can help effectively utilize a smaller TEMP folder.

Note:

• Setting the TEMP folder to a disk separate from the primary OS can optimize resource-intensive workflows. The TEMP folder location can be changed post-installation using the FME_TEMP environment variable.
• Setting the TEMP folder to a network drive is not recommended due to potential performance degradation.

Conclusion

In summary, planning an FME Flow installation demands a careful assessment of your organization's workflows, data volumes, and performance needs. This guide offers key insights into the resource requirements for both Express and Distributed deployments. Tailoring your hardware to support your most demanding processes is essential for optimal performance. By following these guidelines, you can ensure a robust and scalable FME Flow deployment that meets your organization's current and future needs efficiently.