Maximum concurrent FME processes error

Symptom

If you see the following error message, then you have exceeded the limit of 8 concurrent fme.exe processes running with a single FME Desktop license:

Error message was: FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

Note: The setting is system-wide so having multiple installs on the same machine will not allow for more simultaneous processes.

This limit is different from that placed on the parallel processing of "group by" transformers. These transformers limit the number of processes based on license level. More information on this can be found in the product documentation and the blog post FME 2012 Sneak Peek: Parallel-Processing.

Cause

1. Running more than 8 FME processes

FME Desktop licenses are restricted to allow a maximum of 8 fme.exe processes to run at any one time. This includes any running Workspaces and any Data Inspectors, either run through command line or workbench, including those launched from another workspace using the WorkspaceRunner transformer.

2. FME.exe not terminating

If you receive this error but are not attempting to run more than 8 concurrent workspaces then check in your Task Manager for any idle fme.exe processes that have failed to terminate upon earlier job completion.

Idle (0% CPU) FME EXE processes still showing in Task Manager after a job is canceled

Resolution

If FME.exe processes are hanging

• Check the Task Manager / Activity Monitor for any idle fme.exe processes that may have failed to terminate upon job completion and manually shut these down to clear the error.
• If you are using the command line to launch a workspace ensure that there are no enabled Inspector transformers within this workspace - these processes will still launch but since the Data Inspector is not actually opened they will not terminate - thus taking up one or more of your 8 processes.
• If an fme.exe process is continually hanging when a specific workspace is run do the following:
  • Report the issue along with a copy of the workspace through our website: Report a Problem as this should not be happening.

If this is a recurring issue, a workaround is to set up a Python Shutdown Script to end all fme.exe processes after the workspace in question is done. You can find an example script here.

Example Shutdown Python Script

Note: You will not be able to do this if there are other fme.exe processes still running as these will also be terminated, but you could set up a schedule for a script to run at times you know the there are no other running workspaces.

If you are using a WorkspaceRunner

The WorkspaceRunner transformer was developed so that running one workspace would trigger the run of a series of child workspaces. This works by spawning a new FME process and is useful for batch processing however using this transformer can generate the licensing error. You can control the spawning of new FME processes in the transformer parameters to help reduce the risk of this error occurring.

The key parameter to be aware of is ‘Wait For Job to Complete’:

• If this is set to Yes (default) then the transformer will wait for each job to complete before processing the next feature so will not spawn off additional FME processes.
• If this is set to No then the user is essentially allowing parallel processing and two more parameters become available:
  • ‘Maximum Number of Concurrent FME Processes’ - defines the number of FME processes to be launched so that multiple workspaces can run simultaneously. The main workspace uses a single fme.exe process leaving a further 7 processes available to be launched for use by the child workspace. If no value is set then the maximum of 7 concurrent processes will be used.
  • ‘Number of Workspaces per FME Process’ - controls the number of workspaces to be run consecutively on a single FME process before it is shut down. This may help prevent the chance of hanging fme.exe processes.

WorkspaceRunner parameter editor

Adjusting these parameter settings should help you to avoid further encounters with the licensing error. For example: setting the ‘Maximum Number of Concurrent FME Processes’ to 5 would leave 2 FME processes available to run other workspaces.

Why not try FME Server?

If you are interested in implementing a larger number of concurrent processes, we suggest you look into FME Server, which is a scalable solution that allows you to increase the number of concurrent processes based on your needs – by increasing the number of FME Server Engines.

What is FME Server?

FME Server is a web application that works hand in hand with FME Desktop providing an automation platform for all your workspaces.

FME Server supplies two major features that would help you overcome this licensing error:

• Complex Scheduling - Scheduling in FME Server allows for more complex task management than setting workspaces to run through the command line. Advantages include:
  • Automated job submission 24/7 reducing the need for many processes to be run in parallel.
  • Attempt recovery if a job initially fails it will be resubmitted up to three times.
  • Enables job submission in response to other processes. For example, if job X runs successfully run job Y, or Directory/Database watching to trigger a job based on an event.
• Queues
  • In FME Server if a job is run and there are no engines available, rather than the job failing it will join a queue and run once an engine is free.
  • You can create queues with different levels of priority and assign different engines and repositories to them. Therefore a single engine could be allocated to high priority jobs removing the need for these jobs to join the queue behind less significant or urgent jobs.

From WorkspaceRunner to FMEServerJobSubmitter

If you make the transition to run a workspace on FME Server you will need to replace any WorkspaceRunner transformers with the FMEServerJobSubmitter.

The FMEServerJobSubmitter has the same underlying role as the WorkspaceRunner but provides additional parameters such as designating a Job Queue. This transformer submits one job per feature that enters it but key parameters to be aware of here are found under Submission Mode:

• ‘Submits Jobs’ - this transformer determines whether a request is submitted as soon as an initiating feature has arrived (In Sequence) or whether it waits until all input features have been received before sending out the requests (In Parallel).
• ‘Wait for Jobs to Complete’ -
  • If this is set to Yes then the jobs are run synchronously, that is the transformer will wait until the job is completely processed and the initiating feature is output via the Succeeded port with additional attributes added to it.

If this is set to No then the jobs are run asynchronously, meaning the job id is output via the Submitted port as soon as the server receives the request.

FMEServerJobSubmitter parameter editor

Known issues with FME Server 2017 or older

• If you cancel an FMEServerJobSubmitter job before the process completes the corresponding fme.exe might fail to terminate causing subsequent JobSubmitter jobs to fail. This could happen when Wait for Job to Complete = Yes (synchronous) as all child jobs will run on the same engine taken up by the initiating job, and no other engines are engaged. See the resolution to this here.
• Prior to 2017.1 FME Server was behaving in the same manor as FME Desktop whereby the maximum concurrent processes was on a per machine basis. Therefore no matter how many engines your FME Server has only 8 fme.exe processes could be launched at anyone time. More information on this limitation is available here.
• If your workspaces were created in a version pre-2017 please make sure the FMEServerJobSubmitter version is up to date.

For any further troubleshooting help please refer to the article: FME Server Troubleshooting: FMEServerJobSubmitter.