FME Server Workflow Management

If you are using FME Server 2019+ it is recommended that you use Automations for workflow management. Specifically the Job Orchestration article covers this scenario.

Introduction to Workflow Management

FME Server allows you to build up complex workflows by linking individual workspaces together and taking actions depending on whether the workspaces completed successfully.
 

FMEServerJobSubmitter

The FMEServerJobSubmitter is the key transformer for managing FME Server workflows. It is possible to use a controller workspace to run jobs while waiting for them to complete, without tying up another FME Engine on the server. The method described in this article replaces several more complicated methods that we’ve previously recommended. This example requires only one FME Engine. If you are interested in utilizing multiple FME engines, please see FME Server Workflow Management with Multiple Engines.
 

Requirements

• FME Workbench 2017.1
• FME Server 2017.1
• Completed Getting Started with FME Server Series

Downloads

load-data.fmwt

process-data.fmwt

rasterize-data.fmwt

chain-jobs.fmw

• Controller workspace: chain_jobs.fmw
• Workspace 1: load_data.fmwt
• Workspace 2: process_data.fmwt
• Workspace 3: rasterize_data.fmwt

Please download one workspace file (.fmw) and three workspace templates (.fmwt - workspace with data attached)
 

Workflow Management Example

In this demonstration, we will use the example of loading vector data from a shapefile into a database, processing the data, and creating raster web map tiles. This is achieved using 4 workspaces - one controller and three workers. The controller runs each worker workspace in turn, and FME Server could be configured to send an email upon successful completion of the entire process.

Step 1: Building the workflow components

Typically, you would build the individual workspaces which make up your workflow. For this tutorial, we’ve already built the worker workspaces as described above. For details on how they work, open the *.fmwt and *.fmw files in FME Workbench, and read the annotations.
 

Step 1a: Publish process workspaces

Publish all the process templates (listed below) to a new repository on your FME Server. On the second upload screen make sure you select "upload data" and "select a location" repository (Temp should work well in this case). Lastly, you can assign it to the "Job Submitter" Service. If you need specifics on how to upload data and use it in FME Server please see the Getting Started with FME Server Tutorial.

• Workspace 1: load_data.fmwt
• Workspace 2: process_data.fmwt
• Workspace 3: rasterize_data.fmwt

Step 1b: Test process workspaces

Test the workspaces you shouldn't need any configuration changes to test. Just submit the job from the "Run Wokspace" page.

Step 2: Linking the components together using a controller

Once the individual component workspaces are published, you can start setting up the controller workspace (chain_jobs.fmw). For this tutorial, a controller is already built. However, you will need to go through the wizard for each FMEServerJobSubmitter to configure it properly for your FME Server installation (update the FMEServerJobSubmitters with your hostname and credentials), and choose the correct values for the published parameters, however, you should only need to configure the a) Server connection, b) repository, and c) workspace. Everything else should be maintained the same. Please note that the process workspaces should already be published at this point.

The workflow of the controller workspace (chain_jobs.fmw) is started with a Creator. In this case, we use the UUIDGenerator and the AttributeCreator to create a unique directory for each run. These two transformers generate a unique id and append it to a server directory FME_SHAREDRESOURCE_TEMP. Once the chain_jobs workspace has been uploaded to FME Server the parameter FME_SHAREDRESOURCE_TEMP by default automatically maps to the Temp folder in you Web Interface Resource tab.

The FMEServerJobSubmitters are configured as follows:

Step 2a: Configure FMEServerJobSubmitter_1

The first FMEServerJobSubmitter needs the input files which are set in published parameters of the controller workspace. The Terminators are used for exception handling and will log which workspace caused a failure. They will also cause the controller workspace to report failure, rather than reporting a false success.

When writing out to file-based data format (as we are with the SpatiaLite database) you will have to pass the shared directory created with the UUIDGenerator and the AttributeCreator between the job submitters. So your Destination of one FMESeverJobSubmitter and the Source of the following FMESeverJobSubmitter must match.

Advanced settings can be ignored at this time.
 

Step 2b: Configure FMEServerJobSubmitter_2

If you were to use a DBMS, you would instead specify the connection parameters as published parameters of the controller workspace.
 

Step 2c: Configure FMEServerJobSubmitter_3

In the last FMEServerJobSubmitter, we want to write the output (web map tiles) to a location we can access once the jobs are complete. In this case, we chose the temp directory again.

Step 3: Running the entire workflow

We are now ready to publish the controller workspace, chain_job.fmw. It can be published to the Job Submitter service. Once it has completed you can confirm it is successful by looking for the web tiles in the new "UUID"/TileFolder in the Temp Resources folder.

Next: FME Server Workflow Management with Multiple Engines

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver