Introduction
There are a number of factors that need to be considered when you are planning an FME Flow (formerly FME Server) installation. This guide is intended to provide all the resources you need to help you make decisions about how to implement your FME Flow.
Planning your FME Flow Deployment
Architecture
FME Flow consists of multiple components that may be installed together or separately. Understanding each component’s function is critical for planning an effective deployment.
The 5 main components of FME Flow are:
- The Core manages and distributes job requests (queuing, request routing, scheduling), repository contents (workspaces, custom formats, custom transformers, data), and automation processes. Contains a Software Load Balancer (SLB) that distributes jobs to available FME Engines.
- The Engines process job requests by running FME Workspaces. Each FME Engine processes a single request at a time. FME Flow processing can be scaled by adding FME Engines to the same computer or to separate computers within a distributed FME Flow environment.
- The Web Application Server runs the FME Flow Web User Interface, FME Flow Web Services, and any other web clients. The installer provides a default Apache Tomcat, but users may provide their own custom web applications.
- The Database stores all metadata related to FME Flow, including jobs, repositories, automations, users, and other data. The installer provides PostgreSQL by default, but users may provide their own custom database (PostgreSQL, Oracle, or SQL Server).
- The System Share is a directory used to store workspaces files, log files, and data shares. The system share is set at installation and cannot be moved without reinstalling. In a Distributed/Fault-Tolerant Installation, the FME Flow System Share files are installed in directories on a remote file system to enable access from multiple clients.
FME Flow Engines
FME Flow Engines are responsible for processing jobs. Engines may be installed on the same machine as the rest of the FME Flow Components, separately, or both. For best performance, careful consideration should be given to where the Engines are installed and what resources are available to them.
FME Flow supports three different engine types: Standard, CPU Usage, and Remote. All Engine types process jobs the same way, but they differ by billing and installation. To decide which engine is right for your deployment, consider your job duration, frequency, and how the jobs will be triggered.
| Engine Type | Standard | CPU Usage (Dynamic) | Remote Engines |
| Pricing | Purchase a perpetual fixed engine. | Purchase credits for Engine CPU time. No cost for idle engines. | Standard or CPU |
| Flexibility |
Installed alongside FME components Installed to another machine (Distributed) |
Installed alongside FME components Installed to another machine inside your network (Distributed) |
Installed to another machine (Distributed) Distributed to another machine inside your network (Distributed) Distributed to another machine outside your network (Distributed) Distributed to the Cloud (Azure Functions, Google Functions) |
| Scalability |
Launch the number of Engines assigned to your FME Flow license. Additional engines can be purchased through an Account Manager. |
Unlimited CPU Engines, as needed. Limited only by credits and hardware. | Standard or CPU |
| Recommended Usage |
Predictable and consistent workloads. Includes but not limited to:
|
Unpredictable workloads. Includes but not limited to:
|
Predictable or unpredictable workflows. Includes but not limited to:
|
Leverage different Engine types for different processes in your deployment for maximum flexibility.
In all scenarios, your FME Flow Engines should be as close to your data as possible. This reduces the effect of network latency and wait times. Keep this in mind when choosing whether FME Flow should be installed on-premise or in a virtual.
Deployment Options
An FME Flow installation is highly configurable. Each component can be installed together or individually. The ideal architecture will rely on your organization’s workflow, data, access, security, and performance requirements.
The 3 main deployment models are:
- Express: All FME Flow components are installed on the same machine.
- Distributed: FME Flow components are installed across different machines.
- Fault-Tolerant: FME Flow components are installed across different machines with redundancy.
For an in-depth guide to these deployment options, please refer to A Guide to Choosing your FME Flow Deployment Architecture.
Express
Express is the recommended deployment for all FME Flow customers. Express installers deploy all the default FME Flow components to a single machine. Given adequate resources, an Express installation delivers excellent performance with the best reliability.
Distributed
Distributed installers deploy FME Flow components across multiple machines AND/OR configure FME Flow to use custom components (database, application server, etc.) There are many different variations of distributed installations. Distributed installations may just be an Engine installed on a separate machine, others may isolate the database on a separate machine, and so forth.
Distributed installations are available for customers who have specific requirements for specific FME Flow components. Otherwise, they offer no real operational advantage over Express installations.
Fault Tolerance
To ensure that jobs are processed in the event of failure, FME Flow supports configuring fault tolerance throughout the multiple levels of an integrated system. FME Flow provides fault tolerance in the following ways:
- Recovery: Restarting components and jobs when crashes occur. This is achieved through the FME Flow Process Monitor.
- Redundancy: Ensuring there is no single point of failure. Multiple FME Flow installations are installed on separate servers and a third-party load balancer directs traffic to either.
Disaster Recovery
Disaster recovery is designed to be able to regain FME Flow operations and data in the event of a major failure of a data center. Disaster recovery can be incorporated into any of the fault-tolerant architectures. For more information, see: Planning for Disaster Recovery.
Environment Specifications
Operating System (OS)
FME Flow can be installed on Windows or Linux. See FME Platform Technical Specifications for updated OS requirements.
Keep in mind that not all formats are supported on Linux platforms. Consider your data formats and review their requirements in our format matrix, FME Applications and Format Support on Linux.
Linux
If you opt to install FME Flow on a Linux OS, we recommend that your FME Flow Administrator has Linux experience. The Technical Support Team has put together some resources that should be reviewed before proceeding with this setup:
Containerization
FME Flow is available as YAML files for containerized Linux-based environments.
- Docker supports express FME Flow deployments: Getting Started with FME Flow and Docker
- Kubernetes supports distributed FME Flow deployments and those that require high availability. Helm charts are available for deployment: Getting Started with FME Flow and Kubernetes
Cloud
FME Flow can be deployed to the cloud.
- FME Flow Hosted is our Software as a Service (SaaS) deployment, requiring minimal maintenance. See Feature Comparison: FME Flow and FME Flow Hosted for key differences between a standard FME Flow deployment and FME Flow Hosted.
- FME Flow can also be deployed to your own, self-managed cloud environment. Review Deployment Options for FME Flow in the Cloud for more options.
Installation Considerations
Hardware
Hardware requirements for FME Flow are specific to your data and processes. However, the following resources have been put together to try and help make those decisions:
Managing Multiple Environments
FME Flow can be deployed to multiple environments in order to support Continuous Integration/Continuous Delivery (CI/CD). This allows workflows to be developed, tested, and deployed throughout a lifecycle (e.g. DEV > UAT > STAGE > PROD)
Licensing
The license file is stored in C:\ProgramData\Safe Software\FMEFlow\licenses. For help requesting a license, review the instructions in the Documentation. There are two methods available to license FME Flow through the Web UI: Automatic and Manual.
- Automatic licensing is recommended. Use the internet to connect to the FME Flow Web UI and input your FME Flow serial number and customer information. If your environment is locked down, follow the FME Flow Automatic Licensing URLs article for guidance on what outbound connections to allow. To use CPU-Credit Engines, you must be able to use automatic licensing.
- Manual licensing is available for FME Flow environments that are disconnected from the internet. Download a JSON file and email it to codes@safe.com, where a license file will be generated using the information provided. The response is not immediate, and you should prepare to wait up to half an hour to receive the license file that can then be uploaded through the Web UI. CPU-credit Engine licensing is unsupported by manual licensing.
Silent Install Licensing
If you are installing FME Flow by silent installation scripts, it may be licensed via the FME Flow REST API.
- Automatic POST /licensing/request
- Manual POST /licensing/requestfile
Processes, Ports, and Firewall
FME Flow components and processes require communication over a variety of ports. If you have a firewall in place, you’ll need to create inbound/outbound rules for FME Flow to function as expected. For more information, see FME Flow Ports.
Account Permissions
On a distributed/fault-tolerant install, or if you are accessing data stored outside FME Flow System Share, you’ll be required to run FME Flow Services under a service account.
3 different account types can affect security:
- Group managed service accounts (gMSAs) offer the highest security level due to their automated password management and limited permission scope, making them ideal if Flow services run across multiple servers within a domain.
- Domain service accounts (our usual recommendation) provide a balance between manageability and security, suitable if Flow services need defined/limited permission requirements.
- Domain user accounts are flexible but present the most significant security risk due to manual password management and broader scope permissions. Notably, they run the risk of accidentally getting locked out of a user account if/when services attempt to start with a bad password.
For more information on required permissions, see Directory and Account Permissions.
Third-party Licensing and Workspace Dependencies
To make use of Esri-related formats (ArcSDE Geodb, File Geodb, Personal Geodb), Esri software must be installed and licensed on the same machine as your FME Flow Engine Service. For more information, please see: Using FME Flow with Esri ArcGIS Software.
Once FME Flow is installed, depending on the components in use within your workspaces, other provisions may be required for FME Flow. Some common examples include:
- Python Modules
- Custom Coordinate Systems
- Packages
- Web /Database Connections
- Format Dependencies e.g. JDBC Drivers
Optimization
Optimizing FME Flow is an iterative process and requires continuously analyzing your existing setup. Some of the deciding factors for system improvement are:
- Job processing time
- System resource usage
- Number of engines available
- Business requirements
Check out this webinar where we dive into Best Practices for Optimizing Jobs on FME Flow.
What's Next?
After FME Flow has been installed, you may wish to perform some additional configurations.
Comments
0 comments
Please sign in to leave a comment.