Dynamic Workspace Chaining in FME Flow Automations

Introduction

The aim of this article is to demonstrate how you can use some of the advanced functionality in FME Flow (Formerly FME Server) automations to build a workflow that follows the splitter pattern. The splitter pattern involves dividing one message into multiple output messages. Downstream processes will be run once per message. The individual messages usually contain data about a particular object, event, record, etc. 

Splitter image by Enterprise Integration Patterns , under CC-BY-4.0.

In this example, there are four key Automation features that we’ll be making use of:

If you have not already done so, we recommend following Routing Data Between Workspaces in Automations to familiarize yourself with the content router pattern before proceeding with this tutorial.

Step-by-Step Instructions

In this scenario, you are tasked with building an automated workflow that will process incoming work order requests for city assets at a recurring interval. A different workspace is required to process different asset types, and you’d like to receive an email once all the processing is complete for the day. 

This article has been broken down into three parts;

1. Our first goal is to modify a workspace that reads the full list of work orders and checks which requests are still open and awaiting processing. We’ll need to identify and map the assets to the appropriate processing workspace and pass this data to an FME Flow Automations writer for use in the automation.
2. Second, we’ll build an automation that will be set up to take the information from (1) to trigger a dynamic workspace action. The workspace to run will be determined by the output keys received from the Automations writer, demonstrating the splitter pattern. 
3. Finally, we’ll use the split-merge block to collate all the different processing workspaces, enabling us to send a single notification once all the processing is complete.

Part 1: Update the Workspace to Map Work Order Assets to Corresponding Workspace Names

In this section, we’ll modify an existing workspace. The starting workspace is also used in Routing Data Between Workspaces in Automations. This workspace is designed to filter a dataset for a single work order type and then route the work order contents to an automations writer for further processing. We’ll map the assets to the appropriate processing workspace and pass this data to an FME Flow Automations writer for use in the automation.

1. Open TrafficWorkOrdersToProcessStart.fmw

Download the AdvancedAutomationsTutorial.zip attached to this article. Unzip the file and open TrafficWorkOrdersToProcessStart.fmw in FME Workbench.
 

2. Update the Tester

The Tester is currently performing two actions; a) checking for work orders where the status = submitted and b) looking for School Asset Types. In this example, we would like to remove the second test so that we can prepare all work orders for processing.

Open the tester parameters, remove the bottom three rows of the Test Clause and select OK. There should only be the Status = Submitted clause. 

3. Add an AttributeValueMapper
This transformer will be used to map different asset types to the appropriate processing workspace. This information will then be sent to the Automations writer and ultimately used to configure the Dynamic Workspace action in Automations. 

Add an AttributeValueMapper to the canvas; connect the input port to the AttributeKeeper and connect the output port to the WorkOrders writer feature type.
 

Tip: Click on the connection line between the AttributeKeeper and WorkOrders writer feature type before adding the AttributeValueMapper for the connection lines to update automatically. 

Open the AttributeValueMapper parameters, set the Input Attribute to Asset Type and the Output Attribute to  WorkspaceName. Then in the bottom right corner, select ‘Import’.

In the Import Wizard, set the Format to CSV and the Dataset to wherever you have the TrafficWorkOrders.csv file saved.

Select the Parameters button and change the delimiter to a comma ( , ) and click OK. 

Click Next and set Import from to Attribute Values, then click Continue. 

On the Select Attributes page, set the Input Value to Asset Type and leave the Output Value empty. Select Import and you’ll now see all the work order types listed.

Use the following table to set the Output Values and then select OK:

The Output Value is the name of the workspace that needs to be run in response to a specific work order type. For work orders that have not been categorized, the workspace runs a series of validation checks to correctly assign an asset. When the asset is near a school, a workspace will be run to locate the school in question and notify them to highlight the potential risk to students’ safety, the remaining assets simply need to be processed triggering a workspace that will create an inspection request.

4. Save & Publish to FME Flow

Save the workspace. Then on the top menu bar, select File > Publish to FME Flow…

In the Publish to FME Flow wizard, create a new Repository called AdvancedAutomations and update the workspace name to TrafficWorkOrdersToProcess.fmw

Continue through the wizard, and then publish the workspace under the Job Submitter service.
 

Part 2: Run a Workspace Depending on the Work Order Information

In this section, we’ll build an automation that will be set up to take the information from part 1 to trigger a Dynamic Workspace action. The workspace to run will be determined by the output keys sent by the Automations writer.

1. Upload AdvancedAutomationsTutorial.fsproject

In part 1 of this tutorial, we set up an AttributeValueMapper to map the work order asset type to the corresponding workspace name that needs to be run within the automation. This project (available in the zip file attached to this article) contains the three workspaces in question - so we don’t have to publish them individually from FME Workbench.

Open a web browser and navigate to the FME Flow Web UI, then log in to FME Flow. 

On the side menu bar, expand Projects, then select Manage Project. 

On the Projects page, select ‘Import’ and upload the .fsproject attached to this article. Select Preview Import and continue to click through to the Project Import Completed screen.

Confirm the import was successful by going to Workspaces > AdvancedAutomations and you should now see three additional workspaces listed; ValidateCategorizeAsset.fmw, ProcessRecord.fmw, ContactSchool.fmw. 

2. Build an Automation
 To build an automation, expand the Automations drop-down menu, and select Create Automation. 

At the top right of the automation canvas, click Menu > Save As and give your automation a meaningful name. 
 

3. Add a Schedule Trigger 
We would like this automation to be run nightly:
Click on the Trigger component on the automation canvas to open the parameters. 
In the parameters, under Trigger select FME Flow Schedule (initiated).  Ensure the Recurrence is set to Daily, then change the Start Time to 22:30. 

4. Add a Run Workspace Action 
The automation should first run the workspace published in part 1. 

Select the plus icon in the bottom left corner and then click and drag on the Action icon (orange) to add an internal action component to the canvas and connect it to the successful output port of the Schedule. 

Tip: Drop the action on the successful connection port of the Schedule to connect the two nodes automatically. 

Click on the unconfigured Action node on the automation canvas and select ‘Run a Workspace’ as the Action. Set Repository to AdvancedAutomations and Workspace to TrafficWorkOrdersToProcess.fmw.

Point the Source Dataset to:
$(FME_SHAREDRESOURCE_DATA)/AdvancedAutomations/TrafficWorkOrders.csv
 

After clicking Apply, the Run Workspace component will have a third output port ‘WorkOrders’. This port is tied to the Automation writer present in the workspace and will parse features out to be processed in downstream automation actions.

To learn more about how this writer works, see Routing Data Between Workspaces in Automations. 

5. Add a Dynamic Workspace Action
Add a new Action component to the canvas, and connect it to the WorkOrder Automation Writer output port of the Workspace action. 

Click on the unconfigured Action and select ‘Run a Dynamic Workspace’ as the Action, Set Repository to AdvancedAutomations and point the Workspace to use a value from upstream attributes. To do this, click on the drop-down arrow next to Workspace, then expand Workspace > WorkOrders then select WorkspaceName.

This configuration allows us to trigger a different workspace depending on the work order details. 
The workspaces share the same parameters, so select Import Parameters and choose ContactSchool.fmw from AdvancedAutomationRepository.

6. Save the Automation and Test Configuration 

Your automation should now look like the image below: 

Save the automation and in the top left corner click Start Automation. Open the Schedule parameters, then click ‘Trigger’ to kick off the schedule immediately. 

Go to the Jobs tab in the FME Flow Web UI and select Completed to see one TrafficWorkOrdersToProcess.fmw workspace, along with a handful of the three others. 

Part 3: Send Email On Workspace Completion

In this section, we’ll use the split-merge block to collate all the different processing workspaces, enabling us to send a single notification once all the processing is complete. 
 

1. Add Split-Merge Block 
Stop the automation and disconnect the Schedule trigger from the TrafficWorkOrderToProcess.fmw component.

In the automations component menu, find the split-merge block and add it to the canvas.

Move both action nodes into the split-merge block. Connect the Schedule trigger to the input, and connect both the success and failure ports of the dynamic workspace node to the corresponding output ports of the split-merge block.

2. Configure to Send an Email When All Workspaces Finish
Add an Email action to the success port of the split-merge-block and complete the configuration.

Copy the Email action and add it to the failure port of the split-merge-block, and change the Subject/Body as you wish.

If you do not have details to configure an Email action, use the Log Message internal action to send a message to the logger to test the split-merge-block. 

3. Save Automation and Test Configuration
Save the automation, then click Start Automation. Open the Schedule trigger, then click ‘Trigger’ to kick off the schedule right away.
Check your email (or automation log) to make sure you were notified once all the jobs were completed.

Part 4: Enable Automated Retry for Guaranteed Delivery

This section is optional and aims to explore the action retry functionality in automations. In this example, we provided a CSV file as the source dataset; however, it’s more plausible that this information would be stored in a database. Imagine there is a network connectivity issue at the time your schedule is triggered. In this case, the job will fail and the automation will end, it would be preferable for FME to wait a defined period of time and then try running the job again. This can be accomplished by enabling automated retry. 

This functionality is available on all external actions as well as the Run Workspace and Run Dynamic Workspace actions. For more information, see Configuring Guaranteed Delivery in FME Flow Automations with Automated Retries.

1. Configure Run Workspace Action for Retry
Click on the TrafficWorkOrdersToProcess.fmw action to open the parameter editor.

Click on the Retry tab and check Retry on Failure. Default parameters are configured. To modify the defaults, check the ‘Use custom retry settings’ option and use the following configuration:

Click Apply.
 

2. Test Retry Configuration 
Since we are not connecting to a database, we must simulate a failure. To do this:

Go to the Run Workspace action and update the source dataset to point to a file that does not exist. 

Save and start the automation. Manually trigger the automation, wait a minute, and then view the automation log to observe the retry attempts: 

Since the file does not exist, none of the retry jobs will be successful, this exercise is purely to demonstrate this option.

Troubleshooting

FME Flow Troubleshooting: Automations 

Additional Resources

• Dynamic Workspaces: Data Driven Parallel Processing 
• Configuring Guaranteed Delivery in FME Flow Automations with Automated Retries 
• Job Orchestration with Automations
• Getting Started with the Split-Merge Block
• Understanding the Split-Merge Block | Troubleshooting & FAQs 

Data Attribution

The data used here originates from data made available by the Government of Austin, Texas. It contains information available in the public domain.