FME Flow on Docker: Utilizing Engine Assignment and Job Routing

Introduction

Engines can be assigned to queues based on either engine properties, or engine names. Engine properties by default are the OS, the build of FME Flow (formerly FME Server), the license type, etc. Users are also able to add their own engine properties.
Jobs are routed to queues based on user-defined rules. The traditional method of routing by repository is still available, but there is the additional option to route based on workspace statistics. Workspace statistics track runtime information such as peak memory usage, % CPU utilization and more.
See our blog for an introduction to Analyzing Job Statistics in FME Flow.

This is beneficial for containerized deployments of FME Flow, such as Docker and Kubernetes. Engine names are more likely to change in these deployments, so assigning engines to queues based on engine properties is more useful.

Configure FME Flow Engine Service(s)

In order to add user-defined engine properties, an environment variable for the engine properties must be set as a comma-delimited list. By default, an engine will automatically have several properties assigned to it: FME build, engine type, engine (container) name, OS.

This is what the engine service part of the docker-compose.yaml would look like for three engine services, two standard engine services (with different engine properties) and one dynamic engine service:

fmeflowenginestandard_1:
image: 'quay.io/safesoftware/fmeflow-engine:21313'
volumes:
- 'fmeflow:/data/fmeflowdata'
restart: always
depends_on:
- fmeflowcore
environment:
- EXTERNALPORT=${EXTERNALPORT:-443}
- WEBPROTOCOL=${WEBPROTOCOL:-https}
- ENGINEPROPERTIES=engineGroup1
networks:
- database
- web
fmeflowenginestandard_2:
image: 'quay.io/safesoftware/fmeflow-engine:21313'
volumes:
- 'fmeflow:/data/fmeflowdata'
restart: always
depends_on:
- fmeflowcore
environment:
- EXTERNALPORT=${EXTERNALPORT:-443}
- WEBPROTOCOL=${WEBPROTOCOL:-https}
- ENGINEPROPERTIES=engineGroup2
networks:
- database
- web
fmeflowenginedynamic:
image: 'quay.io/safesoftware/fmeflow-engine:21313'
volumes:
- 'fmeflow:/data/fmeflowdata'
restart: always
depends_on:
- fmeflowcore
environment:
- EXTERNALPORT=${EXTERNALPORT:-443}
- WEBPROTOCOL=${WEBPROTOCOL:-https}
- ENGINETYPE=DYNAMIC
networks:
- database
- web

Scaling Engine Services

Once you have successfully edited your docker-compose.yaml you can run

docker-compose up -d 

By default, this will start one engine per engine service.
Later, if you want to add more engine services or engines, edit the docker-compose.yaml again and then re-run that command.

To scale engines for individual or multiple engine services, use this command:

docker-compose up --scale <engineservicename>=<num_of_engines> -d
docker-compose up --scale <engineservicename>=<num_of_engines> --scale <engineservicename>=<num_of_engines> -d

Configure Engine Assignment Rules

Once FME Flow has been deployed, you will see the newly defined engine properties on the engines page in the FME Flow Web UI:



Next, create queues that correspond to the engine properties and how you'd like to route your jobs. In this example, two new queues will be created, Queue1 and Queue2.



Assign engines to the newly created queues on the Engine Assignment Rules tab.
Add a property that matches the engine properties defined in the docker-compose.yaml (you can refer back to the Engines tab to check this). 
Assign this property to a queue:



In this example, there are 3 Engine Assignment Rules:

Configure Job Routing Rules

At the moment, any jobs that are run on FME Flow will be sent to the Default queue, and processed on the dynamic engine.

Job Routing Rules allow you to control which queue processes will process a job. Job Routing Rules can be created based on workspace repository or workspace statistics, and will be evaluated top down.

For this example, there are 3 Job Routing Rules, based on workspace repository:

Running workspaces from different repositories on FME Flow will show them getting processed by the correct queue: