Job Routing Based on Job Priority

Introduction

This article provides instructions for setting up job routing based on job priority in FME Flow [previously called FME Server]. This is especially helpful when you have limited engines on your FME Flow instance and want to ensure that time-sensitive jobs run before other less critical jobs. 

For the purpose of this demo, it is assumed that there is one engine on your FME Flow instance, but your jobs fall into two categories; important and maintenance.  You would like to configure FME Flow to ensure that important jobs are always run first. This will be achieved in this tutorial by - 

• Creating a new Queue with a higher priority than the default queue. 
• Setting a Job Routing Rule to send the high-priority jobs to the new queue. In this example, all important jobs are stored in the same folder so a Repository based rule will be configured. 
• Configuring an Engine Assignment Rule to allocate an engine to the new queue. Engines can be allocated in two ways: 
  • Assign queues based on engine names. This works well in configurations where the engine names do not change (in local or cloud-based installations) but in the case of docker or Kubernetes-based installations, the engine names will change whenever the engine containers change. 
  • Assign queues based on engine properties (e.g. engine hostname, OS, type of engine). 

In this example, the engine will be assigned using the property type ”standard”. Currently, there is only one engine. But this is useful because if additional standard engines are added in the future, ​​the engine assignment rule doesn’t need to be updated.

This article and test workspaces were updated and tested with FME Flow 2023.2 and it was also tested with FME Server 2021.0, 2022.0 and 2022.2.3.
  

Step-by-Step Instructions  

Part 1: Configure the Queues

1. Navigate to “Engine Management” under the “Admin” section and select “Queues” on the FME Flow Web UI. 

2. Click on “Create ” to create a new queue.  

3. Fill in the appropriate queue name, description, and priority and click OK. For this demo, the name is set to “Higher Priority” and priority set to 10(Highest).
   Note: If no queues are set up on FME Flow, all jobs are routed to a Default queue with priority set to 5(Medium). By setting the new queue priority to 10, any jobs submitted to this queue will be run before those waiting in the Default queue. We can preview the priorities of other existing queues in the system by clicking “Preview queue priorities”
    

Part 2: Configure the Job Routing Rules

Note: To follow the remaining steps exactly, download the attached .fsproject and import it to FME Flow. This project contains the Repository “Important Project” along with a series of workspaces that are used in Part 4 to verify the queue configuration. 


 

1. Navigate to “Engine Management” under the “Admin” section and select “Job Routing Rules” on the FME Flow  Web UI. 
2. Click on “Create” to create a new rule and configure all the parameters as follows and then select OK: 

• Name: Important Job Routing
• Type: Repository
• Repositories: <RepositoryThatStoresHighPriorityJobs> (In this demo, it is “Important Project”) 
• Queue: <QueueCreatedInPart1> (In this demo, it is “Higher Priority”)
• Enabled: On
   

Part 3: Configure the Engine Assignment Rules

1. Navigate to “Engine Management” under the “Admin” section and select  “Engine Assignment Rules” on the FME Flow  Web UI.
2. Click on “Create” to create a new rule and configure all the parameters as follows and then select OK: 

• Name: Important Job Engine Assignment 
• Type: Property
• Rule: Select “Standard” from the drop-down menu 
• Queue: <QueueCreatedInPart1> (In this demo, it is “Higher Priority”)
• Enabled: On
• Active Periods: This field will be greyed out i.e. un-editable if there are no active periods created. For more information on how to set up an active period, please see this document. Once an active period is created, you can select the option in the drop-down menu. For this tutorial, we will skip this field.

Part 4: Verify the Configuration

Once the Queue, Job and Engine assignment rules have been set, navigate to the “Overview ” or “Queues” page to check the configuration. The “Higher Priority” queue is now assigned to the one and only engine on our FME Flow instance since it is a standard engine. 





Note that the “Default” queue is no longer assigned to an engine. In this state, any job that is submitted to the “Default” queue will stay in the queued state and never run because there is no engine assigned to process those jobs. 

For this demo, all the jobs are to be processed by the same engine and the two queues are used to control the order in which they are run. To do this, create a new Engine Assignment Rule “Maintenance Job Engine Assignment”. Configure all the parameters as follows and then select OK:

• Name: Maintenance Job Engine Assignment
• Type: Property
• Rule: Select “Standard” from the drop-down menu 
• Queue: <DefaultQueue> (In this demo, it is “Default”)
• Enabled: On
• Active Periods: Not Applicable

Now, navigate to the “Overview” or “Queues” page to confirm that both queues are assigned to one engine. 
 

Testing Queue Prioritization 

To test this configuration, if you have not already done so, please download the attached .fsproject and import it to FME Flow. To follow the remaining steps, you should make sure in Part 2 you set the Job Routing Rule to the ‘Important Project’ Repository. 

The project file “JobRoutingBasedonJobPriority” consists of the following components - 

• 1 workspace with a long processing time [LongProcessingTimeJob.fmw] 

• 1 manually triggered automation called ‘TestingQueueControl’. It consists of the following workspaces-
  • 2 maintenance jobs [MaintenanceJob.fmw and MaintenanceJob2.fmw] 
  • 1 important job [ImportantJob.fmw] 

Note that the workspace “ImportantJob.fmw” is saved in the “Important Project” repository that is being linked to the “Higher Priority” queue. The other two workspaces are saved in the “Samples” repository and are linked to the “Default” queue [by default]. 






 

1. Run the workspace “LongProcessingTimeJob.fmw” on your FME Flow. 

2. While the workspace “LongProcessingTimeJob.fmw” is still running-

3. Click on “Start Automation” and then click the “Manual Trigger” component and the Trigger button to run the automation. 

4. You will see the three included workspaces queued to be processed.  

The “Overview”, “Engines” and “Queues” pages would also show when engines and queues have jobs running and queued.

You can also see the most recently run jobs on a queue/engine or jobs currently running on the engines/queues.

5. Without configuring the higher priority queue, the workspaces would be run in the order they were received. Instead, after the workspace “LongProcessingTimeJob.fmw” has finished processing, go to the completed jobs page to confirm that the higher priority workspace was processed first.  

 

Additional Resources

Did the Queue Priority Order Change in FME Flow