Job Routing Based on Job Metrics

Introduction

This article provides instructions for setting up job routing based on job metrics in FME Flow [previously called FME Server]. There are a number of use cases where this might be helpful, this includes: 

• Routing jobs to CPU-Usage engines if their CPU time (time spent processing) or CPU % is low (when average CPU time is low, that’s a great signal to take advantage of CPU-Usage Engines).  
• Reserving a single engine for long-running jobs
• Separating resource-intensive jobs for a specific engine host.

For the purpose of the demo, it is assumed there is one standard engine and one CPU-Usage engine running on your FME Flow instance. To ensure cost efficiency, CPU-intensive jobs are preferred to be run by the standard engine and all other less CPU-intensive jobs are to be run by the CPU-Usage or the standard engine. This will be achieved by:

• Creating a new queue that will be assigned to the new routing rules.
• Setting a Job Routing Rule to send the CPU-intensive jobs to the new queue by making use of Job Metrics. In this example, a “CPU Time” based rule will be configured. “CPU time” is the amount of processing time of the translation only inside of FME's control, excluding slow database queries, network latency, or slow HTTP requests.
• Configuring the Engine Assignment Rules to allocate engines to the queues. Engines can be allocated in two ways: 

1. Assign queues based on engine names. This works well in configurations where the engine names do not change (in local or cloud-based installations) but in the case of docker or Kubernetes-based installations, the engine names will change whenever the engine containers change. 
2. Assign queues based on engine properties (e.g. engine hostname, OS, engine type).

In this example, the engine will be assigned to the new queue as the “Standard” engine. Using the engine type rather than the name is useful because if additional standard or CPU-Usage engines are added in the future, the assignment rule doesn’t need to be updated.

This article and test workspaces were updated and tested with FME Flow 2023.2 and it was also tested with FME Server 2021.0, 2022.0 and 2022.2.3. However, the test workspaces will need to be substituted with your own for compatibility reasons. 
 

Step-by-Step Instructions 

Part 1: Configure the queues

1. Navigate to “Engine Management” under the “Admin” section and select “Queues” on the FME Flow Web UI. 

 

2. Click on “Create” to create a new queue. 

3. Fill in the appropriate queue name, description and priority and click OK. For our demo, a “High CPU Job Queue” queue with priority set to  6 will be created. Basically, any priority higher than the “Default queue” that has a priority of 5 should work. We can preview the priorities of other existing queues in the system by clicking “Preview queue priorities”.

Part 2: Configure the Job Routing Rules

1. Navigate to “Engine Management” under the “Admin” section and select “Job Routing Rules” on the FME Flow Web UI. 
2. Click on “Create” to create a new rule and configure all the parameters as follows and then select OK: 

• Name: High CPU Job Routing
• Type: Properties 
• Rule: Select from the dropdown “CPU Time > 5 Minutes”
• Queue: <QueueCreatedForHighCPUJobs> (In this demo, it is  “High CPU Job Queue”)
• Enabled: On

Part 3: Configure the Engine Assignment Rules

1. Navigate to “Engine Management” under the “Admin” section and select  “Engine Assignment Rules” on the FME Flow Web UI.
2. Click on “New” to create a new rule and configure all the parameters as follows and then select OK: 

• Name: High CPU Jobs to Standard 
• Type: Property
• Rule: Select “Standard” from the drop down menu 
• Queue: Default and  <QueueCreatedForHighCPUJobs>(In this demo, it is “High CPU Job Queue”)

• Enabled: On
• Active Periods: This field will be greyed out i.e. un-editable if there are no active periods created. For more information on how to set up an active period, please see this document . Once an active period is created, you can select the option in the drop-down menu. For this tutorial, we will skip this field.

Part 4: Verify the Configuration

Once the Queue, Job and Engine assignment rules have been set, navigate to the “Overview” or “Queues” page to check the configuration.  The “High CPU Job Queue” queue is only assigned to the one “Standard” engine on our FME Flow instance and the “Default” queue is assigned to the one “Dynamic” [i.e. CPU-Usage] engine as well as the Standard engine [as shown in the screenshot below].
 

Testing workspace processing 

To test this configuration,  please download this linked .fsproject and import it to FME Flow Please note that it is a large file [~ 9 GB] created to ensure the workspace has the intended CPU time. It might be required for you to run the workspaces a few times prior to setting queue control, in order to create Job Metrics.



The project file “JobRoutingBasedonJobMetrics” consists of the following components - 

• 1 job that consumes less CPU Time [LowCPUIntensiveJob.fmw] 
• 1 job that consumes higher CPU Time [HighCPUIntensiveJob.fmw] 

1. Run the workspace “HighCPUIntensiveJob.fmw” on your FME Flow from the “Samples” repository on your FME Flow. This workspace is a CPU-intensive job. Please note that the job was routed to the “High CPU Job Queue” and run by the Standard engine.

Note: If you cannot see the ‘Queue’ column on your Jobs Completed page, select the  “Customize Columns” button to make it visible.

2. Run the workspace “LowCPUIntensiveJob.fmw” from the “Samples” repository on your FME Flow. This workspace is not a CPU-intensive job. After the translation has finished, go to the Jobs Completed page to confirm that the job was routed to the “Default” queue. Please note that although this job was run by the CPU-Usage engine this time, it would run on the standard or the CPU-Usage engine based on engine availability.

​​​​​​ 

The “Overview”, “Engines” and “Queues” pages would also show when engines and queues have jobs running and queued.





You can also see the most recently run jobs on a queue/engine or jobs currently running on the engines/queues.