Files
-
- 200 KB
- Download
Introduction
This example demonstrates how to use a controller workspace to utilize multiple FME engines. Using this method, you can run a series of jobs on multiple engines and perform downstream processing from within a single workspace.
New features in the FMEFlowJobSubmitter now eliminate the need for the FMEFlowJobWaiter transformer and enable you to use as many engines as are available. In particular, two parameters in the transformer dictate the behavior:
- Waits for Jobs to Complete: This parameter dictates when the features should leave the transformer (e.g., release a feature before it’s confirmed to have run).
- Submit Jobs: This dictates when the transformer submits the jobs. This is the key parameter that allows you to run as many engines as you have when and have Waits for Jobs to Complete set to Yes.
If you have one engine, make sure that Submit Jobs is set to Parallel, and Wait for Jobs to Complete is set to Yes. Otherwise, FME Flow will deadlock, and you will have to stop the queue and jobs manually to free the engine.
Submit Jobs: |
|||
| Parallel | Sequential | ||
| Wait for Jobs to Complete: |
Yes |
Waits for all jobs to enter the transformer and then submits them at once. Sending features out of the transformer when they have all finished. *This will allow FME Flow to use as many engines as available, but if you only have one engine all child jobs will be stuck in queue. |
Waits for the job to complete and feature to be released before submitting the next job. This combination will spin up a sub-engine to process the child workspaces *Use this setting if you want to use one engine for both the child and parent workspace. |
| No | Waits for all jobs to enter the transformer submits them at once, and releases features as soon as they are completed. | Submits a job as soon as it has been received and releases features as soon as they are submitted. | |
The other new feature to note is the Summary Port. You can now use this to supply information to an FMEFlowEmailGenerator so messages can be sent with information about the jobs submitted. The summary port acts as a blocking transformer and will only release a feature when all the jobs are completed with a fail or success status.
The FMEFlowJobWaiter has been deprecated as of FME 2025.0. You can use an HTTPCaller to make the relevant REST API call directly.
Scenarios
Running One Job Multiple Times With Different Input on Multiple Engines
In this scenario involving allengines.fmw, we would use an FMEFlowJobSubmitter with the Submit Jobs as Parallel and Wait for jobs to complete set to Yes.
Click on the screenshot below to see a larger image.
Running Multiple Jobs With Different Inputs on Multiple Engines
This scenario uses the in-parallel-multiplews.fmw to submit jobs to new engines so you can utilize the full potential of FME Flow. There are several assumptions of this workspace:
1) It’s assuming you want to submit different jobs with different parameters
2) You want to be informed when the jobs are completed so you can process something that is dependent on it.
Click on the screenshot below to see a larger image.
A) Multiple FMEFlowJobSubmitters are needed here so the unique parameters and workspaces can be accounted for.
B) The FMEFlowJobSumbitters Submit Jobs parameter must be set as Parallel and Wait for jobs to complete to No. Even though we want to wait to find the status of the jobs, because the FMEFlowJobSumbitters are blocking transformers, they will not let anything else process in the workspace until they have completed the feature.
C) (Optional)The AttributeCreators create an attribute of holding the workspace name so jobs can be filtered accordingly after the FMEFlowJobSubmitter.
D) FeatureHolder: This step allows all the jobs to be accumulated before passing into the FMEFlowJobWaiter. If they were not accumulated, the triggering of the FMEFlowJobWaiter may cause a FMEFlowJobSubmitter to wait until the feature holder has finished, when we want the jobs sent out as quickly as possible.
E) FMEFlowJobWaiter: This will poll the server to find out when the job has been completed and its status. This transformer has been deprecated as of FME 2025.0. Please use the HTTPCaller or OpenAPICaller and use the REST API.
F) (Optional) Next, we find the status of the Jobs and cancel the process if a job has failed.
Dealing with the output (2 options):
G) Aggregator: The aggregator has two functions
1) to accumulate all the features before releasing a feature
2) To reduce the number of features so we can initiate a new process once.
or
H) TestFilter to filter by Workspace, so you can initiate different processes for different jobs.
Next Steps
In order to be notified of problems with the workflow, you can set up the controller workspace (allengines.fmw, and multiplejobsubmittersallengines.fmw) to publish to a topic on failure, and add an email subscription.
You can also automate the initiation of the workspace by using the Notification service - perhaps to watch a directory for changes, or to respond to an incoming email attachment. This way, FME Flow can automatically load new data into your database, process it, create web map tiles and upload them to a hosting location, such as Amazon S3.