Routing Data Between Workspaces in Automations

Introduction

The FME Flow (formerly FME Server) Automations writer was added to enhance the functionality of automations in FME Flow. As such, it is only of use within an automations workflow. The Automations writer allows FME Flow users to:

• Implement enterprise integration patterns
• Send data from within a workspace to other Automations actions
• Build workspaces to run data-driven parallel processing
• Merge data from other systems

The Automations writer has two main use cases: the message router and splitter patterns. 

The message router pattern is for passing information from one event into downstream actions. Suppose you’ve used the Resource or Network Directory Trigger (a.k.a Directory Watch) in automations before. In that case, you have worked with this type of pattern where the trigger passes the detected file path into a workspace action’s input file parameter. The Automations writer is an extension of this functionality, allowing you to pass custom information stored in attributes from inside a workspace.

Message Router Pattern image by Enterprise Integration Patterns, under CC-BY-4.0.

The splitter pattern involves dividing one object into multiple tasks. Downstream processes will run once per message. The individual messages usually contain data about a particular object, event, record, etc.

Splitter image by Enterprise Integration Patterns, under CC-BY-4.0.

In this tutorial, we will be showcasing both design patterns with the Automations writer.

Terminology

Automations Writer Output Port: For FME’s readers and writers, we refer to different source and destination datasets as feature types. For the Automations writer, each feature type is an individual output port on the workspace action within the automation. 

Automations Writer Parameter: In FME, we refer to attributes as what holds information about a feature. When a feature is routed to an Automations writer, its attributes become an ‘output attribute’, which can be referenced in downstream actions.  

To learn more about automation parameters, please refer to the following article and webinar.

Step-by-step Instructions

This tutorial's scenario is that you frequently receive work order requests to fix school traffic lights across the City of Austin. Schools must be notified so a traffic guard can be put on duty until city maintenance crews resolve the issue. You want to automate this process with FME Flow instead of manually sending the notices.

We must create a workspace that reads the full list of work orders, and checks which requests need to be processed. For each asset, another workspace locates the school that it belongs to and prepares its contact information to use in an automation’s email action.
 

Part 1: Create the Workspace

1. Read the List of Work Orders
Our first workspace will use the splitter pattern by creating separate jobs for each work order. The workspace will read the list of work orders and check which orders have not been processed yet.

Create a new workspace in FME Workbench and add a CSV reader for Traffic_Signal_Work_Orders.csv. Before creating the reader, open the Parameters and set the Delimiter Character to a comma (,). Click OK and OK again. 

Run the workspace with feature caching enabled, then inspect the feature cache in Visual Preview. The work order statuses are tracked in the Status column, and the different statuses include Closed, Submitted, Unassigned, Assigned, and Canceled. We’re interested in fulfilling Submitted requests. 

2. Filter the Orders to Submitted Requests

Add a Tester and set the test clause to Status = “Submitted”. Additionally, we only want to process work orders for school beacons and school beacon zones. Add more test clauses to pass features where the asset type is either “School Beacon” or “School Beacon Zone”. The Composite Expression should read 1 AND (2 OR 3). 

Run the Tester and verify that three features have been passed.

3. Only Keep Relevant Attributes
At this point, we have enough attribute information from each feature to process the requests. The second workspace in our automation will receive this data and locate the asset owners as separate jobs. 

Some irrelevant attributes can be excluded from the features. Add an AttributeKeeper and connect it to the Tester Passed output port. Select the following attributes to keep:

• Asset ID
• Asset Type
• Job Description
• Location
• Problem Found
• Service Request #
• Work Needed
• Work Order ID
• Work Type

Once these have been selected, click OK and OK again.

4. Add an FME Flow Automations Writer

Add a new writer and select the “FME Flow Automations” format. In most cases, you don’t need to change any of the FME Flow Automations writer parameters. Set the Feature Type Definition to “Automatic” and click OK. 

In the Feature Type dialog, set the Feature Type Name to “WorkOrders”. The name of the Automations writer feature type will be the name of the output port in the automation. Click OK when done. Connect the AttributeKeeper to the WorkOrders writer feature type.

If you expand the drop-down schema on the Automations writer, you can see all of the attributes that will be routed to downstream actions in the automation. 

5. Publish the Workspace to FME Flow

The splitter workspace is now complete, and we can publish it to FME Flow. Save the workspace as TrafficWorkOrders-SchoolZones.fmw. Connect to your FME Flow, select or create a repository of your choice, upload the CSV file into the Data shared resource folder, and select the Job Submitter service.
 

6. Inspect the Order Processing Workspace

Open the attached ContactSchool.fmwt template workspace. This workspace will process each asset’s information, find the school that owns it, prepare the contact information, and generate an HTML notice to email to the school.

Right-click on User Parameters in the Navigator pane, select "Manage User Parameters", and note the set of parameters for the Work Order ID, Asset Name, Work Needed, etc.

This workspace uses the ParameterFetcher to create attributes from the published parameter values. The Automations writer in TrafficWorkOrders-SchoolZones.fmw supplies these parameter values. The asset location generates a point, which is reprojected to EPSG:2277. 

The GML reader pulls in the location and contact details of all schools in the City of Austin. The NeighborFinder checks which school the asset is closest to with a maximum search distance of 1 mile. If a match isn’t found or rejected, the workspace will fail.

When a match is found, relevant attributes are kept, and an HTML report containing the work order information is generated. The FeatureWriter creates the HTML notice file, and the summary output port releases a feature containing the file's output path. 

We need to output the school contact information in the email, which isn’t included in the HTMLReportGenerator and FeatureWriter output. The outputs from the AttributeKeeper and FeatureWriter are aggregated to merge their attributes into a single feature, which is routed to an Automations writer.

Note: If we don’t use the Aggregator, two features will be routed to the Automations writer and trigger two emails to be sent for the same work order when we only want one.

7. Publish the Order Processing Workspace

Publish this workspace to FME Flow in the same repository as the previous workspace. Make sure to upload the austinSchools GML and XSD file into the Data shared resource folder:

• Select "Upload data files" then click Select Files
• Click Select Location
• Select "Upload to a shared resource folder" and select the Data folder
• Click OK, OK, and Next

Note: FME Flow's Data folder is a referenced FME Flow Parameter - when configuring file paths, this folder can be referenced as $(FME_SHAREDRESOURCE_DATA)

Select the Job Submitter service, then click Publish.

Part 2: Create the Automation

1. Create a New Automation and Add a Trigger

Open FME Flow and create a new Automation.  Add a Trigger to the canvas and set the following parameters:

• Trigger: FME Flow Schedule (initiated)
• Schedule Type: Basic
• Recurrence: Daily
• Start: any date in the future, and set the time for 08:00

2. Add an Action to Run the TrafficWorkOrders-SchoolZones.fmw Workspace

Connect a new Action to the success port of the FME Flow Schedule Trigger and set the following parameters:

• Action: Run a Workspace
• Repository: Select the repository that the workspace was published to
• Workspace: TrafficWorkOrders-SchoolZones.fmw
• Source CSV: "$(FME_SHAREDRESOURCE_DATA)/Traffic_Signal_Work_Orders_Schools.csv"

Click Apply. The WorkOrders Automation Writer Output Port from the workspace should now be visible below the Action.

Save the automation. 

3. Add an Action to Run the ContactSchool.fmw Workspace

Before we add the next Action, we must first create an output folder that will receive HTML notices generated by the ContactSchool.fmw workspace. Open the Resources page from the left-hand toolbar, click on the Data folder, and click New. Name the new folder Work Notices and click Create. 

Return to the Automations canvas. Connect a new Action to the output port of the WorkOrders Automation Writer and set the following parameters:

• Action: Run a Workspace
• Repository: Select the repository that the workspace was published to 
• Workspace: ContactSchool.fmw

For the ContactSchool.fmw action parameters below, perform the following:

• Open the drop-down menu to access available parameters
• Expand the Workspace and WorkOrders sections
• Select the corresponding attribute for that parameter
  • Ex: For the Work Order ID parameter, select the Work Order ID attribute

Once complete, the published parameters for ContactSchool.fmw should look like the screenshot below.

There are two final parameters to set:

• Source Schools GML File: "$(FME_SHAREDRESOURCE_DATA)/austinSchools.gml"
• Output HTML Notice: "$(FME_SHAREDRESOURCE_DATA)/Work Notices"

Click Apply. The Automations canvas should now look like the screenshot below.

With this setup, ContactSchool.fmw will run for each feature that is routed from the “WorkOrder” Automations writer. The published parameter values will be dynamically filled based on the attribute values of each feature.
 

4. Append the Work Order ID to the Filename of HTML Notices

We want to save individual copies of each work order’s HTML notice with the Work Order ID appended to the filename. To do this, we will use the text editor to add a generic filename and an Automation Writer Parameter to the end of our HTML output path. 

Open the menu next to the Output HTML Notice parameter, and click text editor. The text editor is much like the FME Workbench parameter text editor, where you can use attributes within values.

Add a forward slash to the HTML output path and enter a generic filename for HTML notice, such as "WorkOrder-".  From the parameter menu, select Work Order ID, then type ".html" to specify file type.

The resulting path should look like this: 

"$(FME_SHAREDRESOURCE_DATA)/Work Notices/WorkOrder-{route.WorkOrders.Work Order ID}.html"

The work order ID value will now be appended to the filename of each HTML notice, ensuring the files are uniquely named and won’t be overwritten. When done, click Save and Apply.
 

5. Add an External Action to Send Email Notifications to Schools

Lastly, we want to email the notices to each school. In the automation, add an external action and connect it to the “Contact Info” output port. Select the “Email (Send)” action.

All emails sent will be via the SMTP protocol through an external account. Get SMTP Server parameters from your email provider, or use Load Template to fill in the blanks for some common providers.

SMTP Account and SMTP Password are required if the email server requires authentication (this is likely). Enter the details of an account you can check.

6. Configure the External Action Parameters to Use Automations Writer Output Attributes

Below, we’ll use the Automations Writer output attributes in several of the email action parameters.

First, go to the Email To parameter and select the “Email” attribute under “Contact Info”. Note that the recipient's email addresses are not real. Instead of using the key, you can manually enter your own email address to receive the notifications.

Next, go to the Email From parameter and enter an email address. This does not need to be the same address that was specified as the SMTP Account - if users reply to the notification message, they will be redirected to this email address. 

For the email attachment, select the “htmlreport_path” attribute under “Contact Info”.

Take some time to personalize the subject and body parameters for each school by using other Automations writer output attributes.

Note: We also have access to the WorkOrders Automations writer output attributes in the email action. To view all upstream attributes accessible in the current action, go to the output attributes tab and toggle on “Show Upstream Attributes” at the bottom.



7. Run the Automation in Debug Mode

Now that the automation is complete click the "Debug" button on the top-right. Instead of waiting for the Schedule to trigger by itself, you can manually run the automation by opening the Schedule trigger and clicking the "Trigger" button.

In the Automation menu, inspect the Debug Log and Triggered Jobs to see how multiple jobs have been submitted for ContactSchool.fmw. If you've configured the email external action to send emails to an account that you have access to, you should receive an email for each job with the corresponding HTML report.

Conclusion

We’ve successfully created an automation that uses the Automations writer to split jobs and pass data between actions. Schools will automatically receive work order notices every morning after the FME Flow Schedule trigger runs the automation at 8:00 AM. When the automation is triggered, ContactSchool.fmw will run once per work order, and an email will be sent to each of the corresponding schools.

Why did we process the work orders across multiple jobs instead of doing everything in a single workspace? Depending on the workflow you’re performing and the data sources/destinations, following the splitter pattern breaks tasks into smaller pieces. Each task will have its own job and status, follow-up actions can be performed according to the status, and the jobs can be run on multiple FME Flow Engines. 

This can often help to avoid situations where a single feature causes an entire job to fail, forcing you to redo a long-running job. Additionally, it can sometimes be more efficient to distribute work across multiple FME Flow Engines, especially if you’re running hundreds or thousands of tasks. 

For more examples of using the Automations Writer, along with some other advanced Automations functionality, check out this tutorial: Dynamic Workspace Chaining in FME Flow Automations.
 

Additional Resources

Getting Started with Automations
Job Orchestration with Automations
Getting Started with Enterprise Integration Patterns 
Automation Keys: What They Are and Why You Should Use Them (Webinar)
Getting Started with the Split-Merge Block
Dynamic Workspaces: Data Driven Parallel Processing in FME Flow Automations
 

Data Attribution

The data used here originates from data made available by the Austin Independent School District (ISD), Texas and the Government of Austin, Texas. It contains information available to the public domain. Data has been modified for this example.