Introduction
FME Flow can scale both vertically (by increasing resources available to existing host machines) and horizontally (by adding additional machines to your FME Flow deployment) to meet organizations’ growth needs. This article reviews the various methods for scaling FME Flow deployments and includes important considerations for each approach.
Performance Optimization
Optimizing workspaces and engine usage can reduce resource requirements, so we recommend starting here.
One example of optimization is spreading out FME Flow’s workload using schedules and automations to ensure resource-intensive jobs are run during low-traffic periods. This means users submitting on-demand jobs will enjoy shorter queues and less resource competition.
Queue control and active periods can also be used to manage job loads across a distributed deployment, or to dedicate existing engines to certain jobs. For example, ad hoc jobs could have their own dedicated engine during peak hours, ensuring they don’t get stuck behind other jobs.
Scaling Engines
FME Flow’s engines are its most resource-intensive component. Thus, many of the scaling options presented in this article focus on either adding engines or increasing the resources available to them.
For more information on Flow’s components, see this article.
Vertical Scaling: Adding More Engines to an Existing Host
Increasing the number of engines on the same host via Web UI is an easy way to increase the number of jobs that can run concurrently. This option is especially relevant for express installations, where there is only one host machine for all of FME Flow’s components.
It’s important to consider that engines on the same host will share resources, so you will need to make sure the host has enough RAM, CPU, and storage space to run additional jobs on the added engines.
We also recommend that each engine has a dedicated CPU core to maximize throughput.
Horizontal Scaling: Distributed and Remote Engines
Regardless of your FME Flow deployment type, you can add additional engine-only hosts. This allows you to increase the number of engines while making sure the additional engines have their own resources.
You can also separate existing engines onto their own machine(s) to increase resources available to them.
If you would like to place a distributed engine host closer to the source data on a separate network, please consider remote engines instead. This article provides more detailed information about remote engines and distributed engines, their differences, use cases, and implementation guidelines.
Third-Party Dependencies and Queue Control
Some workspaces depend on external software such as ESRI ArcGIS. These dependencies must be installed and licensed on every engine host that might run these workspaces.
Queue control can be used to route jobs to engine hosts with the required dependencies installed and licensed. This means only a subset of engine hosts will need the third-party dependency.
Standard vs CPU-Usage Engines
Whether you choose to scale engines horizontally or vertically, choosing the appropriate engine type(s) for your needs is essential for effective scaling.
Standard and CPU-Usage engines have the same capabilities, but CPU-Usage engines allow you to only pay for CPU time instead of paying for another fixed license. CPU-usage engines are particularly useful for handling increased workloads during peak traffic times, temporary projects that require significant data processing, and data streams where messages are only received at specific times of day.
CPU time credits can be used on an unlimited number of engines, meaning you can add as many engines as you need to use them effectively.
For more information about CPU-Usage engines, please see this article.
Scaling VMs
Vertical Scaling: Increasing Host Resources
If you notice that a host machine’s resources are being maxed out, or if you are seeing resource-related warnings in logs, you may consider increasing the size and resources of FME Flow’s host machines to resolve the problem and improve job execution time.
For more information on host system requirements and how to determine which resources may need to be increased, please refer to these articles.
- Resource Planning Essentials for FME Flow Deployments
- FME Flow Host System Sizing
- VM Sizing for FME Flow in the Cloud
Horizontal Scaling: Distributed Deployments
FME Flow’s engines are its most resource-intensive component. If you are using an express installation or a distributed/fault-tolerant deployment with several FME Flow components on the same host, you can secure more resources and reduce bottlenecks for each of FME Flow’s components by separating out the engines and distributing them across hosts on the same network. See the “Horizontal Scaling: Distributed and Remote Engines” section for more details.
The web application server and the core should never be separated.
With the exception of remote engines, all components should be on the same network, in the same timezone, and as geographically close to each other as possible. You will need to ensure that FME Flow’s components can communicate with each other by verifying that the correct ports are open.
Geo-Distributed Deployments
Having multiple deployments of FME Flow or utilizing remote engines allows you to place FME Flow’s components closer to various data sources, thereby improving performance. Each separate FME Flow deployment will operate independently with its own engines and resources.
Horizontal Scaling: Fault-Tolerant Deployments
High traffic volumes to FME Flow’s web application server, such as with high-volume data virtualization requests, may impact performance.
Choosing a fault-tolerant FME Flow deployment will enable a load balancer to distribute traffic evenly between two web application servers, thereby improving responsiveness. For more information, see this article.
FME Flow’s fault-tolerant architecture, with its redundant components, also minimizes downtime and improves engine scalability.
Scaling with Containerized and Cloud Deployments
For more information on using FME Flow with Jenkins, Docker, Docker Compose, Kubernetes, and IaC, please see this collection of articles.
Non-Containerized Cloud Environments
Engine and/or Core VMs can be scaled up horizontally in response to longer queue lengths in cloud environments, either manually or on a schedule, using tools such as Azure VM Scale Sets and AWS Autoscaling Groups.
We recommend the web application server and core remain on the same VM(s), as seen in our IaC templates.
For information on scaling engines with Azure VM Scale Sets, please refer to this article.
We recommend implementing a network load balancer between the engines and cores if your cores are deployed with Scale Sets or Autoscaling Groups. For more information, please see this article.
We also have tools available for Infrastructure as Code (IaC) deployments, including a proof of concept for scaling CPU-Usage engines.
For more information on IaC deployments, please see this article.
Containerized Environments
Docker and Docker Compose
Our Docker deployment of FME Flow utilizes Docker Compose. It is not a distributed deployment, but a single-node multi-container setup.
The primary way to scale a Docker deployment of FME Flow is to add additional engines, as mentioned here. Adding FME Engines on a Separate Machine that is not containerized is not tested, supported, or recommended by Safe Software.
This article helps with setting engine properties that can be used when assigning engines to queues.
It is also possible to run multiple instances of FME Flow on a single machine using Docker for testing. This is not recommended for production environments.
To increase resources available to FME Flow on Docker, add sufficient CPU and memory resources to the Docker host
For a distributed deployment or more advanced scaling options, consider using Kubernetes.
Kubernetes (K8s)
While Docker provides a containerized FME Flow deployment for simpler use cases, Kubernetes enables a distributed, highly available deployment with orchestration, automatic scaling, and self-healing capabilities.
Like other fault-tolerant FME Flow deployments, Kubernetes can balance traffic loads across core pods (these also include the web application server), ensuring no one pod gets overwhelmed. It also allows auto-scaling. Engine pods can be scaled based on CPU and memory utilization.
When new host nodes, core pods, or engine pods are added, Kubernetes automatically handles the networking and firewall configuration. You can also adjust the number of resources new pods will require. For more info on Why would you use a Kubernetes deployment for FME Flow?, please follow the linked article.
Scaling capabilities are limited to the FME Engines and FME Flow Core/Web Services Pods; scaling is not available for any other Pods in the deployment. For instructions on how to scale the engines, please see this resource.
For help configuring the specific engine properties used to assign FME Flow engines to the appropriate job queues, please see this article.
Deploying with Kubernetes across Multiple Hosts enables the FME Flow Core to be duplicated, which builds resilience into the system. Should any single host fail, the mirrored FME Flow Core instance on a different machine can immediately take over operations, preventing an outage.
Scaling with FME Flow Hosted
FME Flow Hosted instances can be resized as needed by changing the instance type or adjusting disk space. While you can spin up an unlimited number of engines, the size of your Flow Hosted instance will determine the number of engines that can run effectively.
Please note that primary disk space can only be increased, not decreased. And, changing the temporary disk space will delete its contents.
Conclusion
FME Flow offers a wide range of horizontal and vertical scaling options that can be used individually or in combination to meet your growing data processing needs. Options vary based on your FME Flow deployment and differ in complexity.