FME Flow Jobs go straight into ‘Queued’ state and never run

Symptom

When you submit a job to run on FME Flow (formerly FME Server), the job is forever listed under the Jobs > Queued tab:


If you click on the job there is an error message that the log file does not exist:


Note: These images were captured in 2021.1, the interface may be different, but the concepts are the same. 

Cause

There are no Engines available to run the job. Possible reasons include:

• The FME Flow Engines Service is not running
• FME Flow has not been licensed
• The Engines cannot connect
• The Jobs are being submitted to a user-defined Queue that has not been assigned Engines
• The Engines are occupied by other jobs

Please note that this list covers the most common problems but it is not exhaustive. If you have ruled out all of the suggested resolutions below please contact Safe Software Support and provide a copy of your logs folder from <FMEFlowSystemShare>\Resources. 

Resolution

1. Confirm the FME Flow Engines Service is running 

Windows 

1. Open Services (A Desktop App on Windows found through search)
2. Find FME Flow Engines and look under the Status field. If it does not say ‘Running’ right-click on the Service and select ‘Start’.

Linux

1. Go to <FMEFlowDir>/Server/
2. Run ./startEngines.sh

If you are unable to start the engine service, first try restarting all services and also make sure the account running this service has read/write permissions to the FME Flow Install Directory and System Share [Documentation]. 

2. Check FME Flow is Licensed

In the FME Flow Web UI check that FME Flow is licensed. The Licensing page is located under Admin > System Configuration

If FME Flow is unlicensed follow the Request and Install a License documentation to request your license, you’ll need to enter your serial number. If you do not have a license, or do not know your serial number, contact your Safe Software account manager or sales@safe.com.

If the license request fails please follow the FME Server Licensing troubleshooting guide. 

3. The Engines Cannot Connect

There are a number of reasons why the engines may not be able to connect, viewing the fmeserver.log from <FMEFlowSystemShare>\Resources\Logs\Core\Current and the fmeprocessmonitorengine.log from <FMEFlowSystemShare>\Resources\Logs\Engine\Current should give you an indication of the problem. 

Check the Engine Count

Through the Web UI go to the Engines page and under the Hosts panel make sure the number of engines is greater than zero. If it is already a number, try dropping it down to zero and back up again. 

FME Flow is Unlicensed 

If the Engines page is slow to load and then reports FME Flow is unlicensed despite step 2 being ok, this suggests there is a permissions issue accessing the license file. This has been observed on a Linux OS when FME Flow was installed on a port less than 1024, in this case the Web Application will be installed as the root rather than fmeserver user. It is recommended to reinstall FME Flow using a higher port, suggested 8080. 

No Hosts

The Host panel on the Engines page should list all the hostnames where the FME Engines Service has been installed. If they are not present, please check the FME Flow Ports are open. 

Distributed Engines & Firewall

If you have performed a distributed installation and your engines are hosted on a separate machine from the core make sure that a firewall is not blocking access. If you have a strict firewall in place you should set up a dedicated port pool for the engines to connect. Follow the instructions under the Defining an Ephemeral Port Range heading on the FME Flow Ports for instructions on how to set this up. 

Failed to Connect to Job Queue

This can happen when the memurai.exe (FME Flow 2023+ win64) or redis-server.exe (all other FME Flow versions) process does not start. Usually, this is because the queue folder is missing from <FMEFlowSystemShare>\Resources\Logs, or the account running the FME Flow Services does not have permission to write to this folder. Please see this article for more information. 

4. The User-Defined Queue has not been assigned Engines

If your jobs have been successfully running on FME Flow but after setting up a new Queue some jobs are now getting stuck the Queue may not have been correctly defined. 

In FME Flow go to Engine Management > Engines and make sure your Queue name is listed next to at least one Engine, if it is not set this up under the Engine Assignment Rules tab. 

In FME Server 2020 or older go to Licensing & Engines > Engines and make sure your Queue name is listed next to at least one Engine, if it is not, assign the queue to an engine from the dropdown choice box.

5. The Engines are occupied by other Jobs

Each Engine can process one job at a time, if all your engines are occupied, jobs will wait in the queue until one becomes available. To check if there are any Engines available look at the Engines page under the 'Current Jobs' field. This will display a job ID if that Engine is currently running a job. 

If your FME Flow has one engine, and you attempt to run a job containing the FMEFlowJobSubmitter with Submission Mode: In Parallel and Wait for Jobs to Complete: Yes your FME Flow will go into a 'locked' state. In this scenario, the engine will be occupied running the parent job and is waiting for any child jobs to complete before it ends and frees up the engine, however, the child jobs will never run because there are no other engines to process them. Instead, consider alternative configurations: 

1. Set the Submission Mode: In Sequence - this will spin up sub-engine processes temporarily to run the child jobs.
2. Set Wait for Jobs to Complete: No - this will end the parent translation freeing up the engine to run the child jobs. Note, this will mean you cannot do any further processing in the parent workspace based on the child jobs completed. 
3. Use Automations to chain the jobs. You can then use the 'Merge' action to perform any post-processing once all the workspaces have finished running. 

Are you still experiencing issues?

Please consider posting to the FME Community Forum if you are still experiencing issues that are not addressed in this article or contact Safe Software Support and provide a copy of your logs folder from <FMEFlowSystemShare>\Resources.