Run a Workspace When Data Arrives in a Directory

Introduction

In this article, we’ll create an FME Flow automation that monitors a directory for incoming data, reads that data into a workspace when it arrives, and then runs it. In the example below, we will wait for a new shapefile to arrive via upload. We will be monitoring a resource folder within FME Flow, but any network directory to which FME Flow has access can be watched. See the FME Flow documentation for details.

Please note that the if you are importing the attached .fsproject project as a non-admin user you will need to have an Admin grant permissions to the project and contents before you can proceed. Please see Known Issue: Importing a fsproject File Requires an Admin to Grant Permissions to the Project for more information. 
 

Step-by-Step Instructions

1. Create a New Automation
Log into FME Flow and expand Automations on the left-side menu then click Create Automations. You will be presented with a canvas with one un-configured Trigger.



2. Configure the Trigger to Monitor a Directory
On the Automations canvas, double-click the Trigger component to open the parameters. Under Trigger, select Resource or Network Directory (updated).



Click the ellipsis button to the right of Directory to Watch and choose an existing directory or create a new one in the Data folder called DirectoryWatch. 



Uncheck DELETE and MODIFY in the Events to Watch for parameter so that only newly created files will trigger the automation. 



To minimize wait time while you validate the workflow, set Poll Interval to 1 Minute. Click Validate and (if valid) click Apply. If the trigger fails to validate (the green bar in the screenshot below will be red and contain a descriptive error message), it is likely the path specified under Directory to Watch is incorrect; try re-entering that parameter using the ellipsis button.



3. Add a Filter
Configure the Next Action component as a filter. A Filter messages Action tests incoming messages for a specified string and outputs the messages that match to the success port. All other messages are routed to the failure port.
 
Since we want to watch the directory in this automation’s Trigger for Esri shapefiles, first click the drop-down arrow to the right of the Key field, and choose File Path. This setting corresponds to the file.path output attribute of the Resource or Network Directory Trigger. For more information on output attributes, see Working with Automation Parameters and Custom Attributes in an Automation.
 
In the Contains field, type .shp (omit wildcard characters). Click Apply. Only the .shp file is required as a workspace source, and we do not want the shapefile’s sidecar files (.prj, .shx, .dbf, etc.) to separately trigger the automation. The filter tool avoids failed workspace runs by only allowing .shp files to pass through as a source dataset.



If you wanted to add support for .shz or .zip files to your automation, you would place additional filters downstream of the .shp Filter’s failure port.



4. Publish a Workspace
Since we’re watching for shapefiles, we’ll want a workspace that takes shapefiles as input. Download the shpCopier.fmw workspace available in the Files section of this article and publish it to FME Flow with Job Submitter enabled as the Service. For a walkthrough of publishing a workspace to FME Flow, see Publish a Workspace to FME Flow and Run It.
 
5. Add a Run Workspace Action
Downstream of the success port of the Filter Action, configure a Run Workspace Action. For Workspace, navigate to shpCopier.fmw. Next, for Source Esri Shapefile(s) click on the drop-down arrow and expand Directory, then click on File Path. File Path is the same output attribute used when configuring the Filter. 



Now click on the ellipsis next to Destination Esri Shapefile Folder, and choose a destination folder (e.g. "$(FME_SHAREDRESOURCE_DATA)/Output"). Make sure this is distinct from the one you monitor for input, otherwise, you’ll end up with a never-ending automation! Click Apply.





6. Save Your Work and Start the Automation
To save your progress, click on the  Save icon located on the toolbar or through the Automations menu.  Choose a name (e.g. Copy Shapefile Uploads) and add some descriptive tags (optional), then click OK. Finally, click Start Automation in the upper right.





7. Test Your Automation
Download the buildingupdate.zip file from the Files section of this article and unzip the component files. Navigate to the Resource folder your automation is monitoring, then click Upload > Files and navigate to the unzipped buildingupdate.zip component files and upload buildingUpdate.shp. For a walkthrough on how to upload files onto a Resource folder see FME Flow Data Upload: Resource Files . 



Wait one minute, then go back to the Automations menu and click Manage Automations. 



On the Manage Automations page, check the box next to the Copy Shapefile Uploads automation that we just built, then select Actions > View Triggered Jobs.



We should see one job from shpCopier.fmw. Click on the job entry to open the job log, and we will see that this was the original .shp file that the filter passed.



We can also verify that the automation was run successfully by navigating to Resource>Data>Output which was set as the Destination Esri Shapefile Folder when setting up the shpCopier.fmw workspace. You should be able to see a buildingUpdate.shp as well as the sidecar files (.prj, .shx, .dbf, etc.) in the output folder.

Conclusion

Well done. Your automation is now prepared to trigger when any file arrives in the directory you specified and pass all shapefiles along to a workspace.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.