Working with Automation Parameters and Custom Attributes in an Automation | User and Global Keys

Introduction

When developing an automation in FME Flow, an output attribute (formerly output keys) can pass along messages through the automation in the form of JSON-formatted key-value pairs. These output attributes can be used to customize the automation and log important information. For an advanced tutorial on automation parameters (formerly global keys), please see the article Sending Email from Automations.

 

In FME Flow there are three types of attributes:

• Event Attributes (formerly Event Keys): these are the standard attributes that come predefined within most triggers or actions.

• Custom Attributes (formerly User Keys): these are defined by the user and can be changed to prevent overwriting of certain attributes if the same action occurs.

• Automation Parameters (formerly Global Keys): a parameter that is created for the entire automation and can be placed anywhere where parameters are allowed.

 

The loose analogy with FME Workbench is that automation parameters are user parameters, and event attributes are attributes.

Step-By-Step Instructions

In this three-part article, we will cover the three types of output attributes in FME Flow: Event, Custom, and Automation. The automation setup will stay the same, we will just change the attributes.

 

The automation that we will be using in this tutorial watches a directory on FME Flow called AttributeDemo. Whenever a new file is added to the directory, the automation will be triggered. We are only interested when shapefiles are added to the directory, so we are going to run the workspace FilterMessage.fmw, this workspace is looking for files that end in .shp. (Alternatively, we could also filter using the Filter action, but this action doesn’t have attributes to set). Finally, once the workspace has finished running, an FME Flow topic will be notified if the newly added file ended in .shp.

To follow along, ensure that you have imported the Output Attribute Demo FME Flow Project and accompanying data folder which can be downloaded from the Files section of this article. To learn how to import a project into FME Flow, please see the Exporting and Importing Projects documentation. 
 

Please note that the if you are importing the project as a non-admin user that you will need to have an Admin grant permissions to the project and contents before you can proceed. Please see Known Issue: Importing a fsproject File Requires an Admin to Grant Permissions to the Project for more information. 

 

Part 1: Using Output Attributes

Almost every component in an automation will have predefined output attributes, which can be viewed in the log or by using notifications, additionally, they can be used downstream in the automation to add attributes. These output attributes are available on most triggers, some actions, and most external actions. To access these output attributes, open the parameters for a component, and an Output Attributes tab will appear.

 

If you do not see the Output Attributes tab, this trigger or action does not support output attributes.

 

1. Inspect Output Attributes

In our automation, all three components (Resource or Network Directory (updated) trigger (formerly Directory Watch trigger), Run Workspace action, and FME Flow Topic (notify) external action) have the option for output attributes.

 

For each component, double-click on the component to open the parameters window, then switch to the Output Attributes tab. Take note of the available output attributes. Most of the output attributes are different and component-specific, but there are two that are the same for each component: Source and Time.

 

When we run this automation, both Source and Time will be recorded for each component. But, because the next component has the same output attribute, the values of the first Source and Time will be overwritten. So when the values are output in the AttributeDemo topic, the values for Source and Time will be the values for the FME Flow Topic (notify) external action component. Preserving these values will be covered in Part 2.

 

Let’s see how these output attributes work.

 

2. Start Topic Monitoring

To see how these output attributes work, we need to start topic monitoring. On the side menu bar, expand Automations, and then click on Notifications (Classic).

 

Then on the top of the Notifications (Classic) page, click on Topic Monitoring. Once on the Topic Monitoring page, type Attribute Demo into the text bar and then select it. Once Attribute Demo is selected, the topic monitoring will automatically start.

 

Note: The Topic Monitoring page needs to stay open for monitoring to continue; ensure all other FME Flow work is done in another internet browser tab or window.

 

If you do not see Attribute Demo as a topic, double-check that the OutputAttributeDemo project was imported correctly.

 

In FME Server 2019, you can access the Topic Monitoring page from the side menu bar under Notifications.

 

3. Start Automation

In a new internet browser tab or window, open up the Output Attribute Demo - Part 1 Begin automation that was included in the project. This can be accessed by expanding Automations on the side menu bar and clicking Manage Automations.

 

Once in the automation, click on the green Start Automation button in the top right corner. Once the automation has started, this button will turn red.

 

4. Upload .shp File

Either in a new browser tab or this one, browse to the Resources page. On the Resources page, navigate to the Data folder, then to AttributeDemo.

 

In the AttributeDemo folder, click the Upload drop-down and select Files. Browse to where the AttributeDemoData folder was downloaded to, and select one of the files ending in .shp. (For this example, you can ignore any of the other shapefile sidecar files, such as .proj or .shx).

 

5. View Topic

Now that the .shp file has been uploaded to the AttributeDemo folder, our automation will trigger, and the Attribute Demo topic will be notified.

 

Navigate back to the internet browser tab where you had the Topic Monitoring page open. Wait approximately 30 seconds from the time you uploaded the file, as the Resource or Network Directory (updated) trigger has a poll interval of 30 seconds.

 

After 30 seconds, you should see the following message:

 

Each of the parameter-value pairs in the topic message comes from the output attributes in each of the components:

 

The parameter-pairs highlighted in blue are from the Resource or Network Directory (updated) trigger. The parameter-pairs highlighted in yellow are from the Run Workspace action. The parameter-pair highlighted in orange is from the FME Flow Topic (notify) external action. The two parameter-pairs that are highlighted in green are the output attributes that appear in all three components of our workspace, so the time noted here, is the time that the topic was notified.

 

Part 2: Creating and Using Custom Attributes

As mentioned in Part 1, sometimes components have the same output attribute, which can result in the parameter-pair value being overwritten by the last component. To prevent this, or to add additional attributes that might not be available by default, we can create custom attributes.

 

Custom attributes are created in the Output Attributes tab on each of the components. If a component doesn’t have the Output Attributes tab, a custom attribute cannot be assigned.

 

1. Create a Custom Attribute

Open up the Output Attribute Demo - Part 1 Begin automation and stop the automation if it was running from Part 1.

 

We are going to create two time custom attributes to save the time for each component to prevent overwriting.

 

Double-click on the Resource or Network Directory (updated) component to open the parameters, and switch to the Output Attributes tab. Once on the Output Attributes tab, click on the plus sign next to Custom Attributes.

 

 

All custom attributes will have the prefix user. before each attribute so you can quickly tell which ones are custom. Create a new Attribute called Time1, then click on the pen icon next to Value to open the text editor.

 

In the text editor, expand Event and then click on Event Time. Then click OK.

 

We now have a custom attribute that will output the time for the Resource or Network Directory (updated) trigger. Click Apply to save the Custom Attribute.

 

2. Create a Second Custom Attribute

Now we need to create a custom attribute for the Run Workspace action. Double-click on the Run Workspace component and switch to the Output Attributes tab.

 

Click on the plus sign to create a new Custom Attribute. Name this Attribute to Time2 and set the Value to {time}, just like the Time1 Custom Attribute.

 

Click Apply to save the parameters for the Run Workspace component.

 

3. Save and Start the Automation

Save the automation, and then start it.

 

4. Start Topic Monitoring

We need to start Topic Monitoring if you navigated away from the page in Part 1. Open the Topic Monitoring page and type Attribute Demo into the text bar and then select it. Once Attribute Demo is selected, the topic monitoring will automatically start.

 

Note: The Topic Monitoring page needs to stay open for monitoring to continue; ensure all other FME Flow work is done in another internet browser tab or window.

 

5. Upload .shp File

Open a new browser tab or window and browse to the AttributeDemo folder in the Data directory.

 

Once in the AttributeDemo folder, click the Upload drop-down and select Files. Browse to where the AttributeDemoData folder was downloaded to, and select one of the files ending in .shp, ensure that the .shp file is different than the one uploaded in Part 1.

 

6. View Topic

After uploading another .shp file, navigate back to the internet browser tab where you had the Topic Monitoring page open. Wait approximately 30 seconds from the time you uploaded the file as the Resource or Network Directory (updated) trigger has a poll interval of 30 seconds.

 

The new notification message will be very similar to the one in Part 1, except now it will have our customized custom attributes, which are highlighted in the image below:

 

For this automation, the time only varies by a millisecond, but in a larger, more complex automation the time difference can vary greatly.

 

Part 3: Creating and Using Automation Parameters

Automation parameters allow you to reference a value throughout your automation wherever output attributes can be used. These parameters start with the global. prefix so you can quickly tell where the parameter is coming from.

 

1. Create an Automation Parameter

Open up the Output Attribute Demo - Part 2 Completed automation or continue in the automation from Part 2 and stop the automation.

 

On the top menu bar of the automation, click on the globe icon to open the Automation Parameters Editor window.

 

In the Automation Parameters Editor window, click on the plus sign to create a new Automation Parameter. Then set the Automation Parameter to Author, and set the Value to your name.

 

Click OK to save the automation parameter.

 

2. Apply Automation Parameter

Now we can use the new automation parameter anywhere in this automation. Since it is just the value of our name, we only need to use it once.

 

Click on the FME Flow Topic (notify) component to open the parameters. In the parameters, click on the drop-down next to Messages and then expand Global. Once Global is expanded, click on global.Author.

 

Click Apply to save the parameters.

 

3. Save and Start the Automation

Save the automation, and then start it.

 

4. Start Topic Monitoring

We need to start Topic Monitoring if you navigated away from the page in Part 1 or 2. Open the Topic Monitoring page and type Attribute Demo into the text bar and then select it. Once Attribute Demo is selected, the topic monitoring will automatically start.

 

Note: The Topic Monitoring page needs to stay open for monitoring to continue; ensure all other FME Flow work is done in another internet browser tab or window.

 

5. Upload .shp File

Open a new browser tab or window and browse to the AttributeDemo folder in the Data directory.

Note: If you have used all of the shapefiles in the previous parts, select them all and delete them. Once deleted, you can reuse the shapefiles.

 

Once in the AttributeDemo folder, click the Upload drop-down and select Files. Browse to where the AttributeDemoData folder was downloaded to, and select one of the files ending in .shp, ensure that the .shp file is different than the one uploaded in Part 1 or 2.

 

6. View Topic

After uploading another .shp file, navigate back to the internet browser tab where you had the Topic Monitoring page open. Wait approximately 30 seconds from the time you uploaded the file as the Resource or Network Directory (updated) trigger is setup to have a poll interval of 30 seconds.

 

The topic message will come back different than what you have seen in Part 1 or 2.

 

Using the Message box in the FME Flow Topic (notify) component will only display what was added to the message, not all of the event attributes we have seen in Part 1 or 2. This message isn’t very descriptive, so we will need to update it.

 

7. Modify Topic Message

Go back to the browser tab with the automation open, and stop the automation. Modify the FME Flow Topic (notify) component parameters. Next to the Message box, click on the drop-down arrow and select Text Editor.

 

In the Text Editor, we have access to all of the event, custom, and automation parameters, as well as any string elements so we can construct our message to be more informative. When we add an attribute to our message, the message will only output the value of the parameter-value pair, so we need to add an attribute name to each of the values. For example, for our automation parameter we need to add Author: before {global.Author}.

 

Enter in the following message:

Author: {global.Author}, File Path: {file.path}, Job ID: {job.id}, Time Submitted: {job.timeSubmitted}

 

Click OK to save the values in the Text Editor and then click Apply to save the parameters.

 

8. Start the Automation and View Topic

Once again, start the automation, upload a .shp file and view the output message in the browser tab with Topic Monitoring enabled.

 

Your output message should look like the following:

 

9. Add Message to Automation Parameters

Messages and various parameters can be added to automation parameters to be used elsewhere in the automation. To do this, click on the drop-down next to Message and select Add to Automation Parameters.

 

When the Automation Parameters Editor pops up, change the parameter name from formattedMessage to TopicMessage, then click OK to save the message as a parameter. This parameter can then be used anywhere in the automation as needed.

 

 

Additional Resources

[Documentation] Working with Output Keys in Automations Workflows
[Webinar] Automation Keys: What They Are and Why You Should Use Them

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.