Job Routing Based on Engine Properties

Introduction

This article provides instructions for setting up engine allocation to queues based on engine properties in FME Flow [previously called FME Server]. This configuration might be helpful when FME Flow has distributed engine hosts. The use cases include creating Queues for an engine host:

• where an ESRI license might be available. 
• with a specific OS/bit version.
• with better hardware specs.
• with either Standard or Dynamic Engines.

For the purpose of the demo, it is assumed the FME Flow installation is distributed with an additional engine host, which has an ArcGIS license installed. The administrator needs to ensure that jobs containing Esri formats are only routed to the Queue that is assigned to Engines hosted on a particular host so that the translation can run successfully. All other jobs can be run by any engine. This will be achieved by:

• Creating a new Queue that will be assigned to the new routing rules.
• Setting a Job Routing Rule to send any Esri-related processing to the new queue. In this example, a “Repository” based rule is configured and all jobs that use Esri formats will be published to this Repository.  
• Configuring the Engine Assignment Rules to allocate engines to the queues. Engines can be allocated in two ways: 
  • Assign queues based on engine names. This works well in configurations where the engine names do not change (in local or cloud-based installations) but in the case of docker or Kubernetes-based installations, the engine names will change whenever the engine containers change. 
  • Assign queues based on engine properties (e.g. engine hostname, OS, engine type). 

In this example, the newly created job queue will be assigned to the engines using the engine names. 

This article and test workspaces were updated and tested with FME Flow 2023.2 and it was also tested with FME Server 2021.0, 2022.0 and 2022.2.3.
 

Step-by-Step Instructions 

Part 1: Configure the Queues

1. Navigate to “Engine Management” under the “Admin” section and select “Queues” on the FME Flow Web UI.

   

2. Click on “Create” to create a new queue.  

3. Fill in the appropriate queue name, description and priority and click OK. For our demo, an “Esri Jobs Queue” queue with priority set to 6(Medium) will be created. Basically, any priority higher than the “Default queue” that has a priority of 5(Medium) should work. We can preview the priorities of other existing queues in the system by clicking “Preview queue priorities”.

Part 2: Configure the Job Routing Rules

Note: To follow the remaining steps exactly, download the attached .fsproject and import it to FME Flow. This project contains the Repository “Job run on Additional Engine Host” along with a workspace that is used in Part 4 to verify the queue configuration. 

1. Navigate to “Engine Management” under the “Admin” section and select “Job Routing Rules” on the FME Flow Web UI. 
2. Click on “Create” to create a new rule and configure all the parameters as follows and then select OK: 

• Name: Routing to Additional Engine Host
• Type: Repository
• Repositories: <RepositoryThatStoresJobsforAdditionalEngineHost> (In this demo, it is ”Job run on Additional Engine Host”)
• Queue: <QueueCreatedInPart1> (In this demo,  it is “Esri Jobs Queue”)
• Enabled: On
   

Part 3: Configure the Engine Assignment Rules

1. Navigate to “Engine Management” under the “Admin” section and select  “Engine Assignment Rules” on the FME Flow Web UI.
2. Click on “CreateNew” to create a new rule and configure all the parameters as follows and then select OK:

• Name: Additional Engine Host 
• Type: Name
• Engines: Select the engines hosted on the additional engine machine from the drop-down menu. In this demo, it is “MG-9598-69_Engine1” and “MG-9598-69_Engine2”. 
• Queue: Default and <QueueCreatedInPart1>  (In this demo, it is  “Esri Jobs Queue”)

• Enabled: On
• Active Periods: This field will be greyed out i.e. un-editable if there are no active periods created. For more information on how to setup an active period, please see this document . Once an active period is created, you can select the option in the drop-down menu. For this tutorial, we will skip this field.

To add custom properties to the “Rule” drop-down, please follow the instructions below-  

• Click “Create Property” on the “Create Engine Assignment Rule” page.  

• On the “Create Property” window, select Type as “Custom”. 
• Enter a Property exactly as it appears on the Kubernetes .yaml file. For assigning the engines with the name :”standard-group” to the queue, enter Property as “standard-group”

Part 4: Verify the Configuration

Once the Queue, Job and Engine assignment rules have been set, navigate to the “Overview ” or “Queues” page to check the configuration. The “Default” queue is assigned to the two engines running on the FME Flow core host “Host: MGE-9684”. The “Esri Jobs Queue” queue [as well as the “Default” queue] is now assigned to the two engines running on the additional distributed engine host “Host: MG-9598-69”. [as shown in the screenshot below]. 
 

Testing Queue Prioritization

To test this configuration, if you have not already done so, please download the attached .fsproject and import it to FME Flow. To follow the remaining steps, you should make sure in Part 2 you set the Job Routing Rule to the “Job run on Additional Engine Host” Repository. 

The project file “JobRoutingBasedonEngineProperties” consists of the following components-

• 1 job saved in the “Job run on Additional Engine Host” Repository [JobForAdditionalEngine.fmw] 
• 1 job to be run by the server host [JobForServerHost.fmw]. This job can be saved in any repository except the  “Job run on Additional Engine Host” Repository. In this demo, it is saved in the “Samples” repository.

1. Run the workspace “JobForServerHost.fmw” from the “Samples” repository on your FME Flow. Please note that the job was routed to the “Default” queue and was run by either the engine on the server host or the additional engine, depending on availability. 

2. Run the workspace “JobForAdditionalEngine.fmw” from the “Job run on Additional Engine Host” repository on your FME Flow. Please note that the job was routed to the “Esri Jobs Queue” and run by the additional engine host.

The “Overview”, “Engines” and “Queues” pages would also show when engines and queues have jobs running and queued.