Reproducing the Functionality of the FMEFlowJobWaiter

Files

WaitForJobs.fmw
- 100 KB
- Download

Introduction

The FMEFlowJobWaiter transformer was deprecated in FME 2024.1, and was then completely removed from the product in 2025.0. Thankfully, FME Flow authors can recreate this transformer’s functionality through a few different methods. This article will cover three in more detail: Automations that utilize the Split-Merge Block, the FMEFlowJobSubmitter transformer, and workspaces utilizing the HTTPCaller and Flow’s V4 REST API in conjunction with looping.

Option 1: Automations + Split-Merge Block

One option to consider when looking to replace the FMEFlowJobWaiter is to utilize an FME Flow Automation. If the workflow is relatively simple (i.e., one workspace running at a time), chaining workspaces one after the other will accomplish the same goal. However, if the integration requires parallel processing (i.e., more than one workspace running at the same time), then a Split-Merge Block should be used.

The main benefit of the Split-Merge Block in the context of job waiting is that all Run Workspace actions contained within the block will be treated as one event and must all be completed before an output message is routed.

If you are trying to submit jobs to more than one Flow installation, please see Options 2 & 3. Automations using the HTTP Request external action can also be used, but they are not covered in this article.

There are several other benefits to using Automations and split-merge blocks, especially when developing new workflows:

1. Enhanced Workspace Communication - making use of the Automation Writer allows users to route data and parameters between workspaces, which can make job orchestration and downstream data sharing much simpler.

2. Compatibility with Limited Resources - Automations that chain single workspaces together are a great option when dealing with limited resources (such as a single engine).

3. Parent-Child Capabilities - Users can nest split-merge blocks inside each other, meaning that more complex workflows that utilize parent-child jobs can be broken up within the Automation, using the block’s functionality to wait for groups of jobs to finish before proceeding.

4. Streamlined Maintenance - While similar functionality can be achieved within workspaces, split-merge blocks require fewer steps to configure. Automations allow authors to break large, complicated workspaces into shorter workspaces that are easier to manage and debug.

Option 2: FMEFlowJobSubmitter

The FMEFlowJobSubmitter transformer can be a viable option when upgrading a simple workspace to an Automation/looping workspace is not feasible. This transformer functions very similarly to the FMEFlowJobWaiter, however it has the added benefit of the updated V4 Flow REST API, which is able to return much more detailed results.

The “Wait for Jobs to Complete” parameter will ensure that the specified job completes before proceeding, as well as hold the engine running the parent job while the child job is running. Adding more than one FMEFlowJobSubmitter to a workspace means that jobs from different Flow installations can be submitted at different points in the workspace. Troubleshooting failed jobs when using this transformer can take a bit of investigative work (especially in the case of an engine crashing or a network disruption). However, users who are looking for more of a “1:1” replacement for the FMEFlowJobWaiter may want to consider this option.

Option 3: HTTPCaller + Looping

The third option for replacing the FMEFlowJobWaiter is to utilize the HTTPCaller and loop within a custom transformer. This method can be useful in situations where breaking up a large workspace into an Automation may not be feasible, when jobs must be submitted to different FME Flow installations, or when an Automation Run Workspace internal action cannot be used.

The example workspace in the ‘Files’ section of this article demonstrates how this option can work. However, keep in mind that specific workspace behavior (e.g., error message handling) will need to be adjusted based on your needs.

Download the ‘WaitForJobs.fmw’ workspace and open it in FME Workbench.

In this workspace, jobs are submitted to FME Flow via Flow’s V4 API. Then, a second API request retrieves each submitted job’s status on a loop until the response indicates the job has been completed. Once all submitted jobs have been completed, the workspace will continue.

We will cover the workspace's two main sections, 'Job Submission' and 'JobWaiter Replacement,' in more detail.

Job Submission

The transformers in the Job Submission bookmark represent a scenario where multiple jobs are being submitted via FME Flow’s REST API with the HTTPCaller. Here is a breakdown of how this section works:

1. Configure HTTPCaller

At the beginning of the workspace, the creator sends 20 features to the HTTPCaller. For each feature the HTTPCaller receives, it sends a POST request using FME Flow’s V4 API to submit a sample workspace. This results in 20 submitted jobs. Each job is submitted asynchronously to avoid potential timeouts with jobs that wait in a queue for long periods of time or take a long time to run.

2. Add Tester to Find HTTP Status Codes

A Tester checks that each job was successfully submitted by evaluating the HTTP response (200). The IDs of the jobs that were successfully submitted get passed on, while the failed responses are logged.

3. Flatten the JSON Response

A JSONFlattener parses the response bodies from each successful job submission request and gets the job IDs.

After we have determined that these jobs have been submitted successfully, the next section of the workspace will replace the FMEFlowJobWaiter by monitoring each job's status with API requests.

JobWaiter Replacement

This bookmark contains the replacement for the FMEFlowJobWaiter. First, a FeatureHolder waits until all the jobs submitted in the previous section have been queued before proceeding to the custom transformer. Next, a custom transformer called the JobWaitLooper receives each submitted job ID and checks its status.

The job id must be exposed for the custom transformer to work.

Here is a breakdown of each of the transformers in the JobWaitLooper:

1. Configure the HTTPCaller

The HTTPCaller uses the job ID to GET information on each job.

In order for this transformer to be utilized in a loop, the ‘Maximum Number of Concurrent HTTP Requests’ must be set to 1.

2. Flatten the JSON Response

A JSONFlattener takes in each request’s response and pulls out the status associated with each job ID.

3. Test Responses for Job Statuses

A Tester checks the status value from each response. If the job does not have a status of ‘submitted’, ‘queued’, or ‘running’, then it is considered complete and gets routed to the 'Passed' port. If it fails this test, then it is considered incomplete and is routed to the 'Failed' port.

4. Send "Incomplete Jobs" through a Decelerator Transformer

The incomplete jobs are routed to a Decelerator, which waits a few seconds before looping back to the HTTPCaller to check the status again. The completed jobs are passed out of the custom transformer.

Lastly, as each job is completed and passed out of the custom transformer, another FeatureHolder waits until all jobs have been completed before proceeding to the rest of the workspace (in this case, just a logger, but could be additional processing if needed).

Conclusion

All three options above can be used to replace the FMEFlowJobWaiter transformer, and each one has its own advantages. If workflow visualization and workspace communication are important, than the Automation + Split-Merge Block may be the right choice for you. If the goal is to have a comparable transformer that is more of a "1:1" replacement of the FMEFlowJobWaiter, then the FMEFlowJobSubmitter might be the best choice. If you are interested in having full control over your workflow (robust, user-defined error-handling), then the HTTPCaller + Looping option may be for you.

Ultimately, the choice is up to the user. It all depends on your specific needs and how you choose to structure both new and existing workspaces.