Leverage Remote Engine Service in FME Flow

This article delves into Remote Engine Services, introduced in 2023.0, and highlights its use cases, architecture, installation and other features.  

Use Cases

Remote Engine Services offers a significant advantage of effectively scaling FME Flow deployments horizontally and executing jobs on external servers, enabling the core FME Flow setup to remain protected behind a firewall.

FME Flow on-premise customers can have existing job workflows run on-premise and new or Cloud-optimized job workflows during peak or scheduled periods run on Remote Engine Services hosted on FME cloud or other cloud platforms. 

It enables the execution of jobs on remote FME Flow setups, potentially one that is situated closer to your data sources to allow for lower latency. This setup would bypass other Queue Control rules that might be already set up on the core host. This service can be deployed on a separate host and then connected to FME Flow core host within your network, to a core host outside your network accessible via external endpoints, or to the core host on FME cloud and other cloud platforms. 

Esri integrations with ArcPy/ArcGIS Pro require Windows libraries. ArcGIS Server for Linux is not a usable alternative. Users of Linux-OS-based FME Flow Hosted, Docker and Kubernetes deployments can also utilize Remote Engine Services on a Windows VM and connect it to the FME Flow-hosted or containerized instance to workaround Esri-formats’ Windows dependency.

Architecture and Installation 

The service uses REST API for all communication between FME Flow core host and the remote engine host. A publicly accessible endpoint or opening additional ports is not required on FME Flow core [since all the requests originate from the core side]. Only the HTTP/HTTPS port must be opened on the Remote Engine host to be publicly accessible. You can install the remote engine service on multiple environments [on-premise, FME cloud or other cloud platforms] and connect it to the same core host. 

Below is a screenshot of all the remote engine-related API calls in FME Flow REST API V4 which is currently in Technology Preview [This feature is under active development. Expect changes to the current behaviour, and avoid using it in production environments.].This would help integration with an existing CI/CD workflow in your organization.  

The service supports two types of clients: FME Flow and 3rd Party Applications. Rest API Public endpoints can be used for third Party Apps to run job workflows [for example: POST /remoteengine/jobs/submit]

There are also some hidden endpoints for configuration between core and remote engine since the core will be used to develop and deploy job workflows to Remote Engine service.

To obtain the dedicated remote engine service installer, visit the FME downloads page. 

Once you have the installer file, follow the instructions on Installing Remote Engine Services to install remote engine services on your remote server. 

Currently, the remote engine has its own user interface, similar to the FME Flow core, accessible via the remote engine host [IP/FQDN].  However, the menu items are minimal compared to the FME Flow core Web UI. You can log into the UI via the admin user account created during installation. However, user login via another system, LDAP, Azure AD, or SAML user accounts created or imported on FME Flow Core Web UI is not supported.  

It also requires the installation of a PostgreSQL DB instance on the remote engine host that would store job and system metadata.

Similar to an express installation, FME Web Application, Core, Database, and Engine services run on a remote engine service host. These are managed within the Remote Engine independent of coordination by the Flow core host. FME remote engine service is a light version of FME Flow.

After installation, it requires setting up a queue dedicated to the remote engine service connection on the FME Flow core machine and that queue will directly submit all the jobs submitted to it to the Remote Engine Service host. Please follow this document, Remote Engine Services - FME Flow, on how to create a remote engine services connection and associate it with queues. 

The setup supports multiple Standard and Dynamic [CPU Usage] engine licenses, depending on your license purchase and the number you input in the “Standard Engines” and “CPU-Usage Engines” fields. Remote engine does not have a separate licensing webpage and is configured as a part of the license on the Flow core.

You can monitor the engine status from Core’s “Remote Engine Services” webpage [under the Engine management section] as Initializing, Unlicensed, Not Ready, Offline or Ready.

FME Flow will queue jobs when Remote Engine Services are offline [for example during low demand period] and it will submit them for processing once they are online.  

When the jobs are sent to remote engines, FME Flow core seamlessly uploads required contents to the remote engine service for job execution, namely the workspace and data like resources, packages, published parameters with ‘File’ type, web connections, web services, and database connections and related published parameters, embedded transformers and also the newly introduced deployment parameters. 

Out of these FME objects listed above, workspaces, resources, packages, web connections, web services, and database connections can be automatically cached on the remote engine host. Published parameters and Deployment parameters cannot be automatically cached currently. This cache can be manually reset via the Core Web UI or the DELETE API call [screenshot of the API call list included above]. This is helpful when you would want to reconnect the engine to a different FME Flow core installation or troubleshoot an issue by clearing the cache. 

The job log run by the remote engine would be available on the remote engine host and Core’s Completed Jobs webpage provides links to them. 

Installing with FME Flow-Hosted 

You might want to install FME Flow on the Safe Software-managed solution i.e.  FME Flow-Hosted and install the remote engine on an on-prem/on-cloud virtual machine. The remote engine installation, connection creation on the core host and job translation should be the same with Flow-hosted core as compared to an on-prem/on-cloud Flow core. 

However, you will need to independently request a license for the remote engine in this case. 

How is it different from Distributed Engines?

Networking

Distributed engines must be within the same network as the rest of the FME Flow components. It requires several ports to be opened with bidirectional and continuous network traffic between different components of FME Flow. Remote engines, however, are best suited for operations in multiple networks, on-premise & on-cloud combinations or hosts running in different time zones.

Installation

Installing remote engine service is easier than installing a distributed engine.

Service Account

Distributed Engines are recommended to be run by the same service account as the other FME Flow System Services on the core host. Remote engines does not have this requirement. 

Job Recovery

Job recovery in both distributed engines & Remote engines setup is handled by the MAX_FAILED_TRANSACTION_REQUEST_RETRIES parameter in the fmeFlowConfig.txt configuration file. But there is an additional layer that sits on top of job recovery in a fault-tolerant or distributed installation. Node A can tell if a job is running on Node B. If Node B goes down/offline during a job run, Node A detects that and can resubmit the job on a healthy engine. This is because all the components share the same FME Flow system database where the job metadata is stored. In the case of remote engines that do not share the same system DB, if Node B goes down while running a job, Node A will not know about the running job on Node B and will not resubmit it to be run on a healthy engine.

Logging

Logs for job run by a distributed engine can be accessed via the FME Flow Core web UI. But, the job log for a remote engine would be available on the remote engine Web UI only and core UI’s “Completed Jobs” webpage provides links to them.

Remote Engines and Automations

In automation, you can run a workspace on a remote engine by assigning the job to the queue associated with the engine under the Advanced section of the “Run a Workspace” action.  

The “Job completed” entry would display as below- 

Additional Features

• Remote Engine Services can accept workspaces that are registered to run only the Job Submitter Service. They can run jobs submitted from Run a Workspace or Run a Dynamic Workspace automations actions, Schedules, and Workspace Apps.
• Connecting to Remote Engine Services through a proxy is supported. You will find these additional settings under the “Proxy enabled” checkbox. You can also use the FME Flow proxy setup under System Configuration>Network & Email >Proxy. 

Future enhancements 

• All other FME Flow Web Services, except the Job Submitter service, are not supported at this time but are planned for future releases.  
• Remote Engine Services cannot run custom formats or linked transformers at this time.
• A Remote Engine Service can connect with only one FME Flow instance at a time. Simultaneous connections with multiple FME Flow instances are not supported.
• Automatic cache reset/purging for a more dynamic environment could be an option in future releases.
• There is a plan for a single integrated Web UI for the Core and the remote engine installation instead of separate UIs and for all configurations to be controlled from the main Flow Core web UI. 
• FME Flow Parameters are not supported with Remote Engines at this time [especially location-based parameters]. If your workspace references these parameters when it is run from FME Flow, please edit them to hardcode the value instead of using the parameters.

Troubleshooting 

Log directory and its contents for remote engine installation are comparable to the logs for core installation and are stored in the local [remote engine host] FME Flow file share. The file share location is provided during the time of remote engine installation.   

Data uploaded to the remote engine during stored translation can be found stored in the file share [For example-workspaces under repositories folder].