FME Server Workflow Management - Multiple Engines

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2017.x

Introduction

This example demonstrates how to use a controller workspace for utilizing multiple FME Engines. Using this method, you can run a series of jobs on multiple engines and perform downstream processing from within a single workspace.

In 2017.1, we have a couple of new features in the FMEServerJobSubmitter which now eliminates the need for the FMEServerJobWaiter transformer and enable you to use as many engines as are available. In particular, there are two parameters in the transformer which dictate the behavior:

  1. Waits for Jobs to Complete: Parameter to dictate when the features should leave the transformer. (eg. release a feature before it’s confirmed to have run)
  2. Submit Jobs: Parameter to dictate when the jobs are submitted by the transformer. This is the key parameter that allows you to run as many engines as you have when and have Waits for Jobs to Complete set to Yes.

WARNING: If you have 1 engine, Submit Jobs is set to Parallel, and Wait for Jobs to Complete is set to Yes, FME Server will deadlock, and you will have to manually stop the queue and jobs to free the engine.

 

    Submit Jobs:  
    Parallel Sequential
Wait for Jobs to Complete:
Yes
Waits for all jobs to enter the transformer and then submits them at once. Sending features out of the transformer when they have all finished.

 

 

*This will allow FME Server to use as many engines as available, but if you only have one engine all child jobs will be stuck in queue.

Waits for the job to complete and feature to be released before submitting the next job. This combination will spin up a sub-engine to process the child workspaces

 

*Use this setting if you want to use one engine for both the child and parent workspace.

  No Waits for all jobs to enter the transformer submits them at once, and releases features as soon as they are completed. Submits a job as soon as it has been received and releases features as soon as they are submitted.

 

The other new feature to note is the Summary Port. You can now use this to supply information to an FMEServerEmailGenerator so messages can be sent with information about the jobs submitted. The summary port acts as a blocking transformer and will only release a feature when all the jobs are completed with a fail or success status.

 

Scenarios

Running one job multiple times with different input on multiple engines

In this scenario involving allengines.fmw, we would be using a FMEServerJobSubmitter with the Submit Jobs as Parallel and Wait for jobs to complete to Yes.

kqkP0fxAKs_u-LsYF98SdXtV4jVLDj2Xin90yXFMfHcpCN4bnzpquYUwl1cUerDzbPO-5Pfj5a5baIpG7pH4Q6tL3pCr1zAbiKxUQPqWnYsSqBrJAV8IFaC9lplIFCp1OdglzhJP

Running multiple jobs with different input on multiple engines

This scenario uses the multiplejobsubmittersallengines.fmw to submit jobs to new engines so you can utilize the full potential of FME Server. There are several assumptions of this workspace though 1) It’s assuming you want to submit different jobs with different parameters 2) you want to be informed when the jobs are completed so you can process something that is dependent on it.

 

 

 

multeng.png

A) Multiple FMEServerJobSubmitters are needed here so the unique parameters and workspaces can be accounted for.

B) The FMEServerJobSumbitters Submit Jobs parameter must be set as Parallel and Wait for jobs to complete to No. Even though we want to wait to find the status of the jobs, because the FMEServerJobSumbitters are blocking transformers they will not let anything else process in the workspace until they have completed the feature.

C) (Optional)The AttributeCreators create an attribute of holding the workspace name so jobs can be filtered accordingly after the FMEServerJobSubmitter.

D) FeatureHolder: This step allows all the jobs to be accumulated before passing them into the FMEServerJobWaiter. If they were not accumulated the triggering of the FMEServerJobWaiter may cause a FMEServerJobSubmitter to wait until the feature holder has finished when we want the jobs sent out as quickly as possible.

E) FMEServerJobWaiter: This will poll the server to find out when the job has been completed and its status.

F) (Optional) Next we find the status of the Jobs and cancel the process if a job has failed.

Dealing with the output (2 options):

G) Aggregator: The aggregator has two functions 1) to accumulate all the features before releasing a feature and 2) To reduce the number of features so we can initiate a new process one time.

or

H) TestFilter to filter by Workspace so you can initiate different processes for different jobs.

 

Next Steps

In order to be notified of problems with the workflow, you can set up the controller workspace (allengines.fmw, and multiplejobsubmittersallengines.fmw) to publish to a topic on failure, and add an email subscription. See our tutorial series on email in FME Server for more information - specifically, sending an email when a workspace completes.

You can also automate the initiation of the workspace by using the Notification service - perhaps to watch a directory for changes, or to respond to an incoming email attachment. This way, FME Server can automatically load new data into your database, process it, create web map tiles and upload them to a hosting location, such as Amazon S3.

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.