Files
Introduction
This article explains how to route jobs based on queue metrics in FME Flow. Use this setup when FME Flow has both Standard and CPU-Usage engines, and you want jobs to use CPU-Usage engines only when all Standard engines are busy. Before you start, review the Queue Control documentation.
In the following steps, you will create a new holding queue with a higher priority than the Default queue. No jobs are sent to this queue. Next, you will create engine assignment rules that assign engines to queues. Jobs go to the available Standard engines first. CPU-Usage engines are assigned only when jobs are waiting in the queue. When no jobs are waiting, the CPU-Usage engines return to the holding queue, and only Standard engines process jobs.
This article and the test workspaces were last tested with FME Flow 2026.2. Previous test workspaces were tested with FME Flow 2021.0, 2022.0, 2022.2, and 2023.2.
Requirements
- FME Flow with at least one Standard engine and at least one CPU-Usage engine. This example uses two of each.
- CPU-Usage engines require a CPU-Usage license, which may add cost.
- Administrator access to FME Flow to create queues and engine assignment rules.
Step-by-Step Instructions
This tutorial has four parts. In Parts 1 to 3, you create a holding queue and two engine assignment rules, then verify the configuration. In Part 4, you test the configuration with the JobRoutingBasedonQueueMetrics project file attached to this article.
Part 1: Configure the Queues
In this part, you create a holding queue. The queue needs a higher priority than the Default queue, but no jobs are sent to it.
1. Open Engine Management
Log in to FME Flow. Go to Admin > Engine Management.
2. Create a New Queue
Click the Queues tab, then click Create.
3. Configure the Queue
In the Create Queue dialog, set the following parameters:
- Name: Idle Queue
- Description: Holding queue for dynamic engines
- Priority: 7 (Medium)
Any priority higher than the Default queue priority works. To see the priorities of other queues in FME Flow, click Preview queue priorities.
Click OK.
Part 2: Configure the Engine Assignment Rules
In this part, you create two engine assignment rules. The first rule assigns CPU-Usage engines to the Idle Queue. The second rule assigns CPU-Usage engines to the Default queue when jobs are waiting in it.
You can assign engines by engine name or by engine property (for example, hostname, operating system, or engine type). Engine names work well for local or cloud installations, where names do not change. For Docker or Kubernetes installations, use engine properties, because engine names change whenever the engine containers change.
1. Open Engine Assignment Rules
On the Engine Management page, click Engine Assignment Rules.
2. Create the Idle Rule
Click Create. In the Create Engine Assignment Rule dialog, set the following parameters:
- Name: Idle Rule
- Type: Property
-
Rule:
- Operator: AND
-
Value: Dynamic
- Select from the drop-down
- Queue: Idle Queue
- Enabled: On
- Active Periods: None
The queue is the holding queue you created in Part 1.
Click OK.
This tutorial does not use active periods. To learn how to set one up, see the Active Periods documentation. After you create an active period, it appears in the Active Periods drop-down.
3. Create the Default Queue Overload Rule
Click Create again. In the Create Engine Assignment Rule dialog, set the following parameters:
- Name: Default Queue Overload
- Type: Property
-
Rule:
- Operator: AND
-
Value: Dynamic
- Select from the drop-down
Do not click OK yet. You add a queue condition to this rule in the next two steps.
4. Create a Custom Queue Property
In the same dialog, click the + Create Property button in the Rule section. In the Create Property dialog, set the following parameters:
- Type: Queue
-
Property:
- Number of queued jobs
- Default
Click OK.
5. Add the Queue Condition and Save the Rule
Click + Rule, then set the following parameters:
-
Value: Number of queued jobs: Default >
0 - Queue: Default
- Enabled: On
- Active Periods: None
This rule assigns CPU-Usage engines to the Default queue whenever one or more jobs are waiting in it.
Click OK.
Part 3: Verify the Configuration
In this part, you confirm that the CPU-Usage engines are assigned to the holding queue.
1. Check the Engine Assignments
On the Engine Management page, open the Overview or Queues page. Confirm that Idle Queue is assigned to the two CPU-Usage engines.
Part 4: Test the Configuration
In this part, you use the JobRoutingBasedonQueueMetrics project file to test the configuration. The project contains a manually triggered automation named LongProcessingTimeAutomation. The automation runs two workspaces at the same time: LongProcessingTimeJob1.fmw and LongProcessingTimeJob2.fmw. The project also contains a separate workspace named SubsequentJob.fmw.
The queue configuration applies only to jobs waiting in a queue. To test it, the automation runs two long jobs that occupy both Standard engines. New jobs must then wait in the queue. While the automation runs, you run SubsequentJob.fmw and confirm that FME Flow routes it to a CPU-Usage engine.
The automation workspaces use a Creator transformer to create 20 empty features and a Decelerator transformer to delay each feature. The default Delay is 120 seconds per feature, so each job runs for about 40 minutes. To change the processing time, edit the Delay Per Feature (Seconds) parameter value in both actions of the automation.
1. Import the Project
Download the JobRoutingBasedonQueueMetrics project file attached to this article. Import it into FME Flow from the Projects page.
- Allow items to be overwritten if they already exist: Deselect.
2. Trigger the Automation
Go to Automations and open LongProcessingTimeAutomation. Click Start Automation, then click Trigger.
3. Check Running Jobs
Go to Jobs > Running. Confirm that both workspaces are running on the Standard engines. To see which engine runs each job, you may need to click Customize Columns.
Running jobs may take a few seconds to appear. If no jobs appear, refresh the page.
4. Run SubsequentJob.fmw
While LongProcessingTimeJob1.fmw and LongProcessingTimeJob2.fmw are still running, go to Run Workspace and set the following parameters:
- Repository: Samples
- Workspace: SubsequentJob.fmw
- Service: Job Submitter
Click Run. SubsequentJob.fmw runs for 60 seconds.
5. Confirm the Job Routing
Go to Jobs > Running. Confirm that the two LongProcessingTimeJob workspaces run on the Standard engines, and that SubsequentJob.fmw runs on a CPU-Usage engine.
Without queues and engine assignment rules, FME Flow runs workspaces on any available engine, Standard or CPU-Usage. With this configuration, jobs use Standard engines by default. Jobs go to CPU-Usage engines only when no Standard engines are available. In this test, the two LongProcessingTimeJob workspaces occupy both Standard engines. SubsequentJob.fmw waits in the Default queue, so FME Flow reassigns a CPU-Usage engine to the Default queue and runs the job on it.
You can also view recent and running jobs for each queue and engine on the Engine Management page. Green icons show that a job is running. Click an engine to open its job history.