Routing Data Between Workspaces in Automations

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2023.2

Introduction

The FME Flow (formerly FME Server) Automations writer was added to enhance the functionality of automations in FME Flow. As such, it is only of use within an automations workflow. The Automations writer allows FME Flow users to:

  • Implement enterprise integration patterns
  • Send data from within a workspace to other Automations actions
  • Build workspaces to run data-driven parallel processing
  • Merge data from other systems


The Automations writer has two main use cases: the message router and splitter patterns. 


The message router pattern is for passing information from one event into downstream actions. Suppose you’ve used the Resource or Network Directory Trigger (a.k.a Directory Watch) in automations before. In that case, you have worked with this type of pattern where the trigger passes the detected file path into a workspace action’s input file parameter. The Automations writer is an extension of this functionality, allowing you to pass custom information stored in attributes from inside a workspace.

MessageRouter
Message Router Pattern image by Enterprise Integration Patterns, under CC-BY-4.0.


The splitter pattern involves dividing one object into multiple tasks. Downstream processes will run once per message. The individual messages usually contain data about a particular object, event, record, etc. 

Splitter
Splitter image by Enterprise Integration Patterns , under CC-BY-4.0.


In this tutorial, we will be showcasing both design patterns with the Automations writer.

 

Terminology

Automations Writer Output Port: For FME’s readers and writers, we refer to different source and destination datasets as feature types. For the Automations writer, each feature type is an individual output port on the workspace action within the automation.
 

AutomationOutputPort


Automations Writer Parameter: In FME, we refer to attributes as what holds information about a feature. When a feature is routed to an Automations writer, its attributes become ‘output attribute’ which can be referenced in downstream actions. 
 

AutomationWriterParameters
 

To learn more about automation parameters, please refer to the following article and webinar.

 

Step-by-step Instructions

The scenario for this tutorial is that you frequently receive work order requests to fix school traffic lights across the City of Austin. Schools must be notified so a traffic guard can be put on duty until city maintenance crews resolve the issue. You want to automate this process with FME Flow instead of manually sending the notices.

We must create a workspace that reads the full list of work orders, and checks which requests need to be processed. For each asset, another workspace locates the school that it belongs to and prepares its contact information to use in an automation’s email action.
 

Part 1: Create the Workspace

1. Read the List of Work Orders

Our first workspace will use the splitter pattern by creating separate jobs for each work order. The workspace reads the list of work orders and checks which orders have not been processed yet.

Create a new workspace in FME Workbench and add a CSV reader for Traffic_Signal_Work_Orders.csv. Before creating the reader, open the Parameters and set the Delimiter Character to a comma (,). Click OK and OK again.
 

CSVFields
 

Run the workspace with feature caching enabled, then inspect the feature cache in Visual Preview. The work order statuses are tracked in the Status column, and the different statuses include Closed, Submitted, Unassigned, Assigned, and Canceled. We’re interested in fulfilling Submitted requests.
 

TesterPassed
 

2. Filter the Orders to Submitted Requests

Add a Tester and set the test clause to Status = “Submitted”. Additionally, we only want to process work orders for school beacons and school beacon zones. Add more test clauses to pass features where the asset type is either “School Beacon” or “School Beacon Zone”. The Composite Expression should read 1 AND (2 OR 3).
 

TransformerLogic
 

Run the Tester and verify that three features have been passed.


3. Only Keep Relevant Attributes
At this point, we have enough attribute information from each feature to process the requests. The second workspace in our automation will receive this data and locate the asset owners as separate jobs. 

Some irrelevant attributes can be excluded from the features. Add an AttributeKeeper and connect it to the Tester Passed output port. Select the following attributes to keep:

  • Asset ID
  • Asset Type
  • Job Description
  • Location
  • Problem Found
  • Service Request #
  • Work Needed
  • Work Order ID
  • Work Type


Once these have been selected, click OK and OK again.
 

4. Add an FME Flow Automations Writer

Add a new writer, and select the “FME Flow Automations” format. In most cases, you don’t need to change any of the FME Flow Automations writer parameters. Set the Feature Type Definition to “Automatic” and click OK. 

In the Feature Type dialog, set the Feature Type Name to “WorkOrders”. The name of the Automations writer feature type will be the name of the output port in the automation. Click OK when done. Connect the AttributeKeeper to the WorkOrders writer feature type.

If you expand the drop-down schema on the Automations writer, you can see all of the attributes that will be routed to downstream actions in the automation. 

Workbench
 

5. Publish the Workspace to FME Flow

Our splitter workspace is now complete, and we can publish it to FME Flow. Save the workspace as TrafficWorkOrders-SchoolZones.fmw. Connect to your FME Flow, select or create a repository of your choice, upload the CSV file into the Data shared resource folder, and select the Job Submitter service.
 

6. Inspect and Publish the Order Processing Workspace

Open the attached ContactSchool.fmwt template workspace. This workspace will process each asset’s information, find the school that owns it, prepares the contact information, and generates an HTML notice to email to the school.

SchoolContact

Open the parameter manager and note the set of parameters for the Work Order ID, Asset Name, Work Needed, etc.

This workspace uses the ParameterFetcher to create attributes from the published parameter values. These parameter values are supplied by the Automations writer in TrafficWorkOrders-SchoolZones.fmw. The asset location is used to generate a point, and it’s reprojected to EPSG:2277. 

The GML reader pulls in the location and contact details of all schools in the City of Austin. The NeighborFinder checks which school the asset is closest to with a maximum search distance of 1 mile. If a match isn’t found or rejected, the workspace will fail.

When a match is found, relevant attributes are kept and an HTML report is generated containing the work order information. The FeatureWriter creates the HTML notice file, and the summary output port releases a feature containing the output path of the file. 

We need to output the school contact information in the email, which isn’t included in the HTMLReportGenerator and FeatureWriter output. The outputs from the AttributeKeeper and FeatureWriter are aggregated to merge their attributes into a single feature, which is routed to an Automations writer.

Note: If we don’t use the Aggregator, two features will be routed to the Automations writer and trigger two emails to be sent for the same work order when we only want one.

Publish this workspace to FME Flow in the same repository as the previous workspace. Make sure to upload the austinSchools GML and XSD file into the Data shared resource folder. Select the Job Submitter service and publish.

 

Part 2: Create the Automation

1. Create a New Automation

Open FME Flow and create a new Automation.  Add a Trigger to the canvas and set the following parameters:

  • Trigger type: FME Flow Schedule (initiated)
  • Schedule Type: Basic
  • Recurrence: Daily
  • Start: any date in the future, and set the time for 08:00


FlowScheduleDetails

2. Configure the Work Order Parameters in ContactSchool.fmw
Connect a new workspace action to the FME Flow Schedule trigger and select the ContactSchool.fmw workspace. For the Source GML File and Destination HTML file parameters, specify the path to the uploaded austinSchools.gml file and an output path for the HTML report. Once complete, click Apply.

SourceAndOutputPaths
 

Click and drag the arrow on the “WorkOrders” output port to the ContactSchool input port. Now, ContactSchool has access to the Automations writer parameters coming from the “WorkOrders” output port. We will be assigning these attributes to parameters in ContactSchool.
 

AutomationSetUp

Open the ContactSchool action parameters. For each published parameter except the source GML File and destination HTML file, perform the following: 

  • Open the drop-down menu on the right side to access available parameters.
  • Expand the Workspace and WorkOrders sections. 
  • Select the corresponding attribute for that parameter.
    • Select Work Order ID for the Work Order ID parameter

SetParameters

Once complete, your work order-related published parameters should look like the screenshot below.
 

AllParametersSet

With this setup, ContactSchool.fmw will run for each feature that is routed from the “WorkOrder” Automations writer. The published parameter values will be dynamically filled based on the attribute values of each feature.
 

3. Append the Work Order ID to the Filename of HTML Notices

We want to save individual copies of each work order’s HTML notice. Open the menu next to the Destination HTML File parameter, and click text editor. The text editor is much like the FME Workbench parameter text editor, where you can use attributes within values.

For the HTML output path, position your text cursor at the end of the HTML file name just before “.html”, and select Work Order ID from the parameter menu. The resulting path should look like this: 

$(FME_SHAREDRESOURCE_DATA)/Work Notices/WorkOrder-{route.WorkOrders.Work Order ID}.html

FileNameEditor
 

The work order ID value will now be appended to the filename of each HTML notice, ensuring the files are uniquely named and won’t be overwritten. When done, click OK and Apply.
 

4. Send Email Notifications to the Respective Schools

Lastly, we want to email the notices to each school. In the automation, add an external action and connect it to the “Contact Info” output port. Select the “Send an email” action.

We’ll use the Automations writer parameters in several of the email action parameters. First, go to the Email To parameter and select the “Email” key under “Contact Info”. (Note that the recipient email addresses are not real. Instead of using the key, you can manually enter your own email address to receive the notifications.) For the email attachment, select the “htmlreport_path” key under “Contact Info”.

image4.png

Take some time to personalize the subject and body parameters for each school by using other Automations writer parameters.

Tip: Notice how we also have access to the WorkOrders Automations writer parameters in the email action. To view all upstream attributes accessible in the current action, go to the output attributes tab, and toggle on “Show Upstream Attributes” at the bottom.

ShowUpstreamAttributes

5. Run the Automation

Now that the automation is complete click the "Start Automation" button on the top-right. Either wait for the Schedule to trigger by itself or manually run the automation by opening the Schedule trigger and clicking the "Trigger" button. In the Automation menu, inspect the Automation Log and Triggered Jobs to see how multiple jobs have been submitted for ContactSchool.fmw. If you've configured the email external action, you should receive an email for each job with the corresponding HTML report.
 

Conclusion

We’ve successfully created an automation that uses the Automations writer to split jobs and pass data between actions. Schools will automatically receive work order notices. When running the automation, ContactSchool.fmw will run once per work order and an email will be sent to each of the corresponding schools.

image12.png

Why did we process the work orders across multiple jobs instead of doing everything in a single workspace? Depending on the workflow you’re performing and the data sources/destinations, following the splitter pattern breaks tasks into smaller pieces. Each task will have its own job and status, follow-up actions can be performed according to the status, and the jobs can be run on multiple FME Flow Engines. 

This can often help to avoid situations where a single feature causes an entire job to fail, forcing you to redo a long-running job. Additionally, it can sometimes be more efficient to distribute work across multiple FME Flow Engines, especially if you’re running hundreds or thousands of tasks. 

For more examples of using the Automations Writer, along with some other advanced Automations functionality, check out this tutorial: Dynamic Workspace Chaining in FME Flow Automations.
 

Additional Resources

Getting Started with Automations
Job Orchestration with Automations
Getting Started with Enterprise Integration Patterns 
Automation Keys: What They Are and Why You Should Use Them (Webinar)
Getting Started with the Split-Merge Block
Dynamic Workspaces: Data Driven Parallel Processing
 

Data Attribution

The data used here originates from data made available by the Austin Independent School District (ISD), Texas and the Government of Austin, Texas. It contains information available to the public domain. Data has been modified for this example. 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.