FME Server Scalability

Liz Sanderson
Liz Sanderson
  • Updated


FME Server provides easy and effective scalability solutions to ensure high performance:
  1. Scaling FME jobs: Add FME Server engines to process jobs concurrently.
  2. Scaling web requests: Web Application Server considerations.
  3. Scaling the database repository: Use FME Server’s PostgreSQL database for your repository, or configure your own database on Oracle or SQL Server.

Add Engines

When it comes to scaling FME Server, the Number 1 consideration is FME Engines. Increasing the number of engines available will give you the most benefit for scaling the throughput of FME Server. You can easily scale FME Server to support a higher volume of jobs, either by adding additional standard engines to the system architecture, or in FME 2020.0+, adding dynamic engines.

The FME Server Core contains a Software Load Balancer (SLB) that distributes jobs to the FME Engines. This single active Core is all you need to scale processing capacity. Each FME Engine can process one job at any one time, so if you have 10 engines, you can run 10 jobs simultaneously! If you plan on running a lot of job requests then your best bet is to increase the number of engines available - this will reduce processing time because more jobs can be processed.

If you have jobs consistently sitting in the Job Queue, or you need faster processing of any backlogs of job requests, you should consider adding more engines to FME Server. Keep in mind that adding more engines will not reduce the time a single translation takes to run. This will be more dependent on the underlying hardware and the makeup of the workspace being run. Complex workspaces, big data manipulation, and large datasets just take more time to run.

Standard and Dynamic Engines have the same processing capabilities, for every standard engine you add you must purchase a fixed engine license, alternatively to work with dynamic engines you must purchase engine credits where one credit equals one hour of engine CPU time, allowing you to launch any number of additional engines. The method by which you add FME Engines will likely depend on the reasons you need to scale your FME Server upon which you can  evaluate which option is most financially viable for your needs.  

(Note: The next sections are really not needed to properly scale FME Server but we've included them for your information.)

Web Application Servers

FME Server’s Web Application Server can process in excess of 100,000 HTTP requests per hour. In environments where an extremely large number of requests are expected, it is recommended to keep things simple by using the single Web Application Server. The FME Engines will remain the bottleneck as even the FME Server core will have no issues with the high number of requests. We believe there is no added benefit for additional FME Server Web Applications.

Use Your Own Database Server

You can choose to create the FME Server Database on the default PostgreSQL database server that is provided. Alternatively, and for added flexibility and scalability, you can create the repository on your own Oracle or SQL Server database server. Keep in mind that the scalability of FME Server will be limited if you don't separate out the FME Server Repository Database.

This is offered as a consideration and there may be other benefits for your Administration team for doing this. Learn about the different options in the FME Server documentation.

Defining Job Routing and Priorities

When a job is submitted, job directives can be passed along with it to control where and when translation occurs.

FME Server 2021.x and newer

Queue Control: For FME Server 2021.0 and newer work has been completed to provide a more advanced interface to manage your FME Server load by defining Engine Assignment and Job Routing Rules. Queues can be created and given priority. Engines are assigned to these queues based on Engine properties. Jobs are then assigned to queues either based on their Repository or Job Metrics. Job Routing Rules can also be prioritized. To learn more, see Queue Control

FME Server 2018.x to 2020.x

Job Queues: In FME Server 2018.0+, Job Priority was deprecated in favor of Job Queues are a mechanism for sending jobs to a specific FME Engine. Multiple engines can be assigned to a Job Queue to help control job distribution. Job Priority is set on the Job Queue, whereby jobs in higher-priority queues may execute before jobs in lower-priority queues. Entire repositories can be assigned to a Job Queue, or an individual workspace can be assigned to the queue when the job is being submitted and will then run on assigned engines with the specified Job Priority. To learn more, see Job Queues.

Using the "tag" directive (or "tm_tag" using the API, Web Services, or Console), you can set engines to process certain jobs based on the tag of the transformation request. To learn more, see Job Directives.

FME Server 2017.x and Older

Job Routing: For FME Servers older than 2018.0, Job Routing controls or spreads out the workload of engines running workspaces. In a distributed environment, you may have a mix of OS platforms where certain FME formats can and cannot be run. For instance, consider an FME Server on a Linux OS. Linux cannot run some of the formats that may be required by your business. It may then be necessary to have a Windows OS configured with an additional FME Server Engine. To learn more, see Configuring Job Routing.

Job Priority: For FME Servers older than 2018.0, FME Server also offers the ability to set the job priority using the "priority" directive (or "tm_priority" using the API, Web Services, or Console). Any jobs sent with a priority tag can be moved higher in the job queue. To learn more, see Job Directives.

Was this article helpful?



Please sign in to leave a comment.