Job Routing Based on Queue Metrics

Introduction

This article provides instructions for setting up job routing based on queue metrics in FME Flow [previously called FME Server]. This is especially helpful when you have standard and CPU-Usage engines set up and want to route jobs to CPU-Usage engines only when all standard engines are occupied. 

For the purpose of the demo, it is assumed that there are two Standard and two CPU-Usage engines on the FME Flow installation. Jobs to be processed will be directed toward the available standard engines and the CPU-Usage engines will only be assigned if there are jobs waiting in the queue. If there are no jobs in the queue, the CPU-Usage engines will drop off and only standard engines will be used This will be achieved in this tutorial by: 

• Creating a new Queue with a higher priority than the default queue. It will be a holding queue, so no jobs will be directed to this queue.
• Configuring the Engine Assignment Rules to allocate engines to the queues. Engines can be allocated in two ways: 
  • Assign queues based on engine names. This works well in configurations where the engine names do not change (in local or cloud-based installations) but in the case of docker or Kubernetes-based installations, the engine names will change whenever the engine containers change. 
  • Assign queues based on engine properties (e.g. engine hostname, OS, engine type). 

In this example, the CPU-Usage engines will be assigned to the idle (newly created) queue by default but they will automatically get re-assigned to the “Default” queue when the “Default” queue has a job waiting at any time. This task also involves creating a new custom Engine Assignment Rule Property. 

This article and test workspaces were updated and tested with FME Flow 2023.2 and it was also tested with FME Server 2021.0, 2022.0 and 2022.2.3.

Step-by-Step Instructions  

Part 1: Configure the Queues

1. Navigate to “Engine Management” under the “Admin” section and select “Queues” on the FME Flow Web UI. 

2. Click on “Create” to create a new queue.   

3. Fill in the appropriate queue name, description and priority and click OK. For our demo, an “Idle Queue” queue with priority set to “6 (Medium)” will be created. Any priority higher than the “Default” queue with a priority of 5 (Medium) should work. We can preview the priorities of other existing queues in the system by clicking “Preview queue priorities”.  

Part 2: Configure the Engine Assignment Rules

1. Navigate to “Engine Management” under the “Admin” section and select  “Engine Assignment Rules” on the FME Flow Web UI.
2. Click on “Create” to create a new rule and configure all the parameters as follows and then select OK: 

• Name: Idle Rule 
• Type: Property
• Rule: Select “Dynamic” from the drop-down menu 
• Queue: <HoldingQueueCreatedinPart1> (In this demo, it is “Idle Queue”)
• Enabled: On
• Active Periods: This field will be greyed out i.e. un-editable if there are no active periods created. For more information on how to set up an active period, please see this document. Once an active period is created, you can select the option in the drop-down menu. For this tutorial, we will skip this field.

3. Click on “Create” to create another new rule and configure all the parameters as follows and then select OK: 

• Name: Default Queue Overload 
• Type: Property
• Rule: Select “Dynamic” from the drop-down menu AND Click “+ Create Property” and configure all the parameters as follows and then select OK:
  • Type: Select “Queue” from the drop-down menu 
  • Property: Select a queue property = “Number of queued jobs” and select a queue = “Default” from the drop-down menu

• Queue: Default
• Enabled: On
• Active Periods: Not Applicable

Part 3: Verify the Configuration

Once the Queue and Engine assignment rules have been set, navigate to the “Overview” or “Queues” page to check the configuration. The “Idle Queue” queue is now assigned to the two CPU-Usage engines of the FME Flow instance [as shown in the screenshot below].  

Testing Queue Prioritization 

To test this configuration, if you have not already done so, please download the attached .fsproject and import it to FME Flow. 

The project file “JobRoutingBasedonQueueMetrics” consists of the following components - 

• 1 manually triggered automation called ‘LongProcessingTimeAutomation’ . It consists of 2 workspaces with long processing time [LongProcessingTimeJob1.fmw and LongProcessingTimeJob2.fmw] running concurrently. This queue configuration only applies to jobs that are waiting in the queue. Therefore, to test the setup, two workspaces need to be triggered that will occupy both the standard engines, forcing the upcoming jobs to be ‘waiting’ in the queued state. This is accomplished by the workspace with a long processing time. Note that the default value for “Delay Per Feature (Seconds)” is set at 120 secs. Please feel free to edit this field to increase or decrease the job processing time.

• 1 job that runs after the automation is triggered [SubsequentJob.fmw] 

1. Trigger the automation  “LongProcessingTimeAutomation”-

2. Click on “Start Automation” and then click the “Manual Trigger” component to run the automation.   

3.    You will see the two included workspaces running by the standard engines.  

4. While the workspaces “LongProcessingTimeJob1.fmw and LongProcessingTimeJob2.fmw” are still running, run the workspace “SubsequentJob.fmw” on your FME Flow.
5. Without configuring the queue and engine assignment rules, the workspaces would be run by any engine on the installation, standard or CPU-Usage. Instead, the first two workspaces “LongProcessingTimeJob1.fmw and LongProcessingTimeJob2.fmw” are only run by the two standard engines. In the event of a queued job and no available standard engines, “SubsequentJob.fmw” will be processed by one of the CPU-Usage engines.

The “Overview”, “Engines” and “Queues” pages would also show when engines and queues have jobs running and queued.



You can also see the most recently run jobs on a queue/engine or jobs currently running on the engines/queues.