Search

Getting Started with Queue Control

Merline George
Merline George
  • Updated

Introduction 

This tutorial series provides instructions for setting up Queue Control on FME Flow [previously named FME Server]. 
In general, a job queue contains a list of jobs waiting to be processed by a subsystem. Jobs are held here until a number of conditions are met and the job is selected for processing. 

In FME Server 2021.0 and the newer FME Flow versions, the job queueing mechanism was expanded to provide much greater control and flexibility in how FME Flow processes jobs and uses the available engines. Now, instead of simply creating queues and assigning engines directly to those queues, you can create engine assignment rules, associated active periods [new feature in 2023.2] and job routing rules to dynamically assign engines to queues and send jobs to specific queues based on criteria you set. 

In FME Flow 2023.2, Engine Management pages in the Web Interface have been redesigned to provide a simplified UX and improved control (and visibility) of the configurations made. Some other features that have been added are -
 

  • A new “Overview” page that provides a summary of queues, engines, hosts and streams. They are displayed in different panels with different sorting options.A font colour scheme has been incorporated to differentiate between standard versus CPU usage engines. We also have coloured icons representing the status of running/waiting/idle queues and engines.

 

unnamed.png

 

  • “Queues” page has improved search, sorting and filter [by Job status and Engine type] capabilities. Queue Priority has been changed from 1 being the lowest priority [highest in prior FME versions] and 10 being the highest [lowest in prior FME versions]. There is also now a feature for batch edit for queue priorities. The side panel [accessed by clicking the individual queue] displaying queue statistics like number of successful, failed, cancelled jobs, average and total wait and run time, has been introduced with an option to reset these statistics.

 

unnamed (1).png

 When creating a queue, you can preview the queue priority of the already configured queues.

unnamed (2).png

  • The updated “Engines” page shows Standard and CPU-usage engines with new sorting and filtering options. The side panel [accessed by clicking the individual engine] displays more information like the most recent jobs, jobs currently running and engine properties.
     

unnamed (3).png

  • The new “Host” page can be accessed from the “Hosts” panel on the “Overview” page by clicking “View All”. 
     

    unnamed (5).png

 

Terminology

In FME Flow, Queue Control is made up of the following components that work together to ensure jobs are routed to the correct engines for processing: 

  • Job Routing Rules: These rules determine which queue any submitted job should be sent to. There are two types of Job Routing Rules that can be configured - 

    • Properties:  rule(s) based on one of these properties or a combination of these job-specific properties- processing time [CPU and Elapsed], system CPU usage [in %], peak memory usage, job repository, related workspace, source type, source name, user name, or user role.

    • Repository: rule(s) based on the repository that a workspace is stored in.
       

  • Engine Assignment Rules: These rules control which engine(s) are assigned to each queue. There are two types of Engine Assignment Rules that can be configured-

    • Property: Properties fall under three categories; Custom, Engine, or Queue. Custom properties are user-defined values assigned to engines; they are commonly used in Docker/K8S deployments. Engine properties are pre-populated based on the attributes of the engine host, these include physical memory and processor count. Queue properties hold values about the jobs already waiting in that queue, for example, the number of queued jobs and the time for all queued jobs to finish.

    • Name: rule(s) based on the name of the engine.
       

In FME Flow 2023, to assist in balancing engine load, a new feature has been added to set a limit on the number of queues to which engines can be assigned. For more information on this, please follow the documentation linked.

 

unnamed (6).png

 

  • Queues: Queues hold jobs sent to them by job routing rules and direct them to engines based on engine assignment rules. Every queue is set with a priority ranging from 1 to 10, where 10 is the highest priority. During installation, a “Default” queue is auto-created with a priority of 5(Medium). Being a middle value, it provides the option to create queues with either higher or lower priority than the default.

  • Active periods: Optionally, if you would like to specify periods of time during which engine assignment rules are active, Active periods allow you to do so. You can specify a start date/time, end date/time and also have recurrence options.  If an engine assignment rule is assigned to any active periods, the rule applies to the associated engines only during those periods. Any job submitted to a queue that does not have an available engine to run it, due to an active period that has not yet started, remains in the queue until the active period begins. For more information on how to create an active period, please follow our documentation . 


FME Server Job Processing without Queue Control

 By default, no engine assignment rules or job routing rules are created/ setup by the system. All jobs to be processed will be routed to the Default queue [it is automatically created during installation] with a priority 5(Medium). The jobs are considered equal and will be run by any available engine in the order in which they were submitted. 

 

Articles

Please be advised that these instructions are based on FME Flow 2023.2. It was also tested with FME Server 2021 and 2022.2.3. Please note that Active Periods and Queue Limit Per Engine are new features in the 2023 version.  If you are working with an older version of FME, you should refer to the version-specific documentation to learn more about what queue functionality was supported. 
 

Please follow the links below to setup queue control on your FME Flow Web UI -

Job Routing Based on Job Priority 

Set up time-sensitive jobs to be processed with higher priority than the jobs with less urgency.

Job Routing Based on Job Metrics 

Setup jobs with high CPU/Memory utilization to be routed to engines with sufficient hardware/ system resources. 

Job Routing Based on Engine Properties 

Setup jobs to be routed to specific engines based on the type of engines [standard or CPU-Usage] or engine host’s name or resources.

Job Routing Based on Queue Metrics 

Setup jobs to be routed to specific engines based on the number of jobs waiting in the queue. 

 

Additional Resources

Queues and Data Processing Jobs: FME Server and the Art of Queuing
Guide to Manage Esri Jobs using Queue Control
Defining FME Engines, Queue Control, and Streams to a Kubernetes Deployment
Did the Queue Priority Order Change in FME Flow

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.