Job Routing Based on Engine Properties

Introduction

This article explains how to set up engine allocation to queues based on engine properties in FME Flow. This configuration may help when FME Flow distributes engine hosts. The use cases include creating Queues for an engine host:

• where an ESRI license might be available. 
• with a specific OS/bit version.
• with better hardware specs.
• with either Standard or Dynamic Engines.

For the demo, assume the FME Flow installation includes an additional engine host with an ArcGIS license installed. The administrator must ensure that jobs containing Esri formats are routed only to the queue assigned to Engines hosted on a particular host so the translation can run successfully. Any other jobs can run on any engine. This will be achieved by:

• Creating a new queue and assigning it to the new routing rules.
• Setting a Job Routing Rule to send any Esri-related processing to the new queue. In this example, a “Repository” based rule is configured, and all jobs that use Esri formats will be published to this Repository.  
• Configure Engine Assignment Rules to allocate engines to queues. Engines can be allocated in two ways: 
  • Assign queues based on engine names. This works well in configurations where engine names do not change (in local or cloud-based installations), but in Docker- or Kubernetes-based installations, engine names change whenever the engine containers change. 
  • Assign queues based on engine properties (e.g., engine hostname, OS, engine type). 

In this example, the newly created job queue will be assigned to the engines using the engine names. 

Step-by-Step Instructions 

Part 1: Configure the Queues

1. Navigate to “Engine Management” under the “Admin” section and select “Queues” on the FME Flow Web UI.

2. Click on “Create” to create a new queue.  

3. Fill in the appropriate queue name, description, and priority, and click OK. For our demo, create an “Esri Job Queue” queue with priority set to 6 or 7 (Medium). Basically, any priority higher than the “Default queue” that has a priority of 5(Medium) should work. We can preview the priorities of other existing queues in the system by clicking “Preview queue priorities”.

Part 2: Configure the Job Routing Rules

To follow the remaining steps exactly, download the attached .fsproject and import it to FME Flow. This project contains the Repository “Job run on Additional Engine Host,” along with a workspace used in Part 4 to verify the queue configuration. 

1. Navigate to “Engine Management” under the “Admin” section and select “Job Routing Rules” on the FME Flow Web UI. 

2. Click on “Create” or “Create Job Routing Rule” to create a new rule and configure all the parameters as follows, and then select OK: 

• Name: Routing to Additional Engine Host
• Type: Repository
• Repositories: <RepositoryThatStoresJobsforAdditionalEngineHost> (In this demo, it is ”Job run on Additional Engine Host”)
• Queue: <QueueCreatedInPart1> (In this demo,  it is “Esri Job Queue”)
• Enabled: On

Part 3: Configure the Engine Assignment Rules

1. Navigate to “Engine Management” under the “Admin” section and select  “Engine Assignment Rules” on the FME Flow Web UI.

2. Click on “Create” or “Create Engine Assignment Rule” to create a new rule and configure all the parameters as follows, and then select OK:

• Name: Additional Engine Host 
• Type: Name
• Engines: Select the engines hosted on the additional engine machine from the drop-down menu.
• Queue: Default and <QueueCreatedInPart1>  (In this demo, it is  “Esri Job Queue”)

• Enabled: On
• Active Periods: This field will show no options if there are no active periods created. For more information on setting up an active period, see the Active Periods documentation. Once you create an active period, you can select it from the drop-down menu. For this tutorial, we will skip this field.

Note: For Kubernetes deployments of FME Engines, You can create a Custom Property with properties from the applicable engine deployment, such as name or engineproperties [as values set up in the Kubernetes .yaml file], and use it to assign engines to queues. For more information, see Defining FME Engines and Queue Control Properties to a Kubernetes Deployment. Please see below an example of a .yaml file for a Kubernetes deployment-

To add custom properties to the “Rule” drop-down, please follow the instructions below-  

• Click “Create Property” on the “Create Engine Assignment Rule” page.  

• On the “Create Property” window, select Type as “Custom”. 
• Enter a Property exactly as it appears on the Kubernetes .yaml file. For assigning the engines with the name :”standard-group” to the queue, enter Property as “standard-group”

Part 4: Verify the Configuration

Once the Queue, Job, and Engine assignment rules have been set, navigate to the “Overview ” or “Queues” page to check the configuration. With this instance, the “Default” queue is assigned to the two engines running on the FME Flow core host “Host: AH-DIST-CORE”. The “Esri Job Queue” queue [as well as the “Default” queue] is now assigned to the two engines running on the additional distributed engine host “Host: AH-DIST-ENGINE”. [as shown in the screenshot below].

Testing Queue Prioritization

To test this configuration, if you haven't already, download the attached .fsproject and import it into FME Flow. To complete the remaining steps, make sure you set the Job Routing Rule to the “Job run on Additional Engine Host” Repository in Part 2. 

The project file “JobRoutingBasedonEngineProperties” consists of the following components-

• 1 job saved in the “Job run on Additional Engine Host” Repository [JobForAdditionalEngine.fmw] 
• 1 job to be run by the server host [JobForServerHost.fmw]. You can save this job in any repository except the “Job run on Additional Engine Host” Repository. In this demo, it is saved in the “Samples” repository.

1. Run the workspace “JobForServerHost.fmw” from the “Samples” repository on your FME Flow. Please note that the job was routed to the “Default” queue and ran on either the server host engine or the additional engine, depending on availability. 

2. Run the workspace “JobForAdditionalEngine.fmw” from the “Job run on Additional Engine Host” repository on your FME Flow. Please note that the job was routed to the “Esri Job Queue” and run by the additional engine host.

The “Overview”, “Engines”, and “Queues” pages would also show when engines and queues have jobs running and queued.