Files
Introduction
This article has two parts. The first will demonstrate how to upload data to Dropbox at the end of an FME Flow automation workflow. The second will cover how to trigger an automation workflow when new data becomes available in Dropbox.
Requirements
- Dropbox account (create a free personal account)
Step-by-step Instructions
Part 1: Upload Files to Dropbox
In this workflow, we will create an automation in FME Flow that monitors a resource directory using the Resource and Network Directory (updated) Trigger and copies incoming data into a Dropbox directory using the Dropbox Directory (upload) External Action.
We will also be configuring a Dropbox account with an app to generate an OAuth 2 access token, which will allow us to access the Dropbox directory.
1. Create a New Automation
From the FME Flow web interface, go to Automations > Create Automation to create a new automation.
2. Configure the Trigger to Monitor a Directory
On the Automations canvas, double-click the Trigger node. Under Trigger, select Resource or Network Directory.
Click the ellipsis button to the right of Directory to Watch and create the path Resources/Data/Automations/exercise data/Dropbox Exercises/toUpload.
Create this exact path, as it is referenced in the attached workspace used in Part 2 of this tutorial.
Since we are monitoring for any incoming data and making a copy available in Dropbox, select Yes for both the Watch Subdirectories and Watch Folders parameters. Watch for CREATE and MODIFY events, but not DELETE. We want to make all the most up-to-date data products available in our Dropbox, but we don’t need to exactly mirror the contents of the monitored resource. For easier automation testing, set Poll Interval to 1 minute.
Click Validate and (if valid) Apply. If the Trigger fails to validate (a red bar containing a descriptive error message will appear), it is likely the path specified under Directory to Watch is incorrect; re-enter that parameter using the ellipsis (...) button.
3. Add an External Action
Downstream, configure a Dropbox Directory (upload to) External Action. If you are in guided mode, simply click on the silhouetted Next Action node and scroll down to External Actions. Otherwise, drag an External Action node out onto the canvas and connect it to the Resource or Network Directory Trigger. Before we can proceed, you'll need to configure your Dropbox account with an app to generate an OAuth 2 access token.
4. Configure Dropbox Account
In Dropbox, create a new app called FME Flow Access that has full access.
Once the app is created, navigate to the Permissions tab, select the following permissions, then click submit:
- files.metadata.write
- files.content.write
- files.content.read
If you leave your Dropbox app page, you can return any time by navigating back to developers.dropbox.com, clicking App console, and selecting the app.
Your OAuth 2 access token is separate from your access key and secret key. Select Settings after selecting the app, scroll down to OAuth 2, and click Generate under Generated Access Token. Copy the token.
5. Configure Dropbox Directory External Action
Navigate back to the automation in FME Flow. In the Dropbox Directory External Action parameters, paste the token into the Access Token parameter.
Note: When permissions are changed, a new access token needs to be generated from the Dropbox application and then added to FME Flow, the change will not be passed on to FME Flow.
Populate Source Path with any files created or modified in the watched directory by clicking the drop-down arrow and then selecting Directory > File Path. Choosing a directory will copy the contents of that directory and all subdirectories to your Dropbox.
Fill in Destination Path with a folder in your Dropbox where you wish to deposit the files. For example, if you want files copied to the root of your Dropbox, simply type “/”. If the folder you specify does not exist, it will be created the first time data is uploaded. Uploads will overwrite existing files with the same name in the Dropbox.
When you are satisfied with your configuration, click Validate and Apply.
6. Save and Start the Automation
Click Menu above the Automations canvas, then Save As. Choose a name and add some descriptive tags, then click OK. Click Start Automation in the upper-right.
7. Test the Automation
Navigate to the Resource folder your automation is monitoring (Resources/Data/Automations exercise data/Dropbox Exercises/toUpload is suggested above), then click Upload > Files and navigate to any file you wish to upload. Wait for at least one polling interval (1 min), then go to Automations > Manage, check the box next to the automation you just built, and select Actions > View Log File.
Since this automation didn't run any workspaces, there will be no triggered jobs, but the Automation log file will include details on which files and file actions triggered a workflow and where they were stored in the target Dropbox.
FME Flow logged the file upload, but we can always verify its success by logging into the target Dropbox and checking the folder path (/FMEFlow/watcherUploads/) we specified for the file uploaded to FME Flow earlier.
Well done! Your automation is now prepared to upload files to the Dropbox folder you specified whenever a file is created or modified in the folder FME Flow is monitoring. Next, we’ll monitor a folder in Dropbox and trigger a workflow in FME Flow when files arrive.
Part 2: Run a Workspace When Data Arrives in a Dropbox
In Part 2 of this exercise, we will imagine that users upload KML files of features digitized in Google Earth to a Dropbox, and we want to monitor that Dropbox for new KML uploads, convert the KMLs to GML, and reproject them for use with other project data.
1. Create a Dropbox Folder to Monitor
In the Dropbox web interface, create a new folder to monitor for this exercise. Below, /FMEFlow/toWatch/ has been created.
2. Configure and Publish a Workspace to FME Flow
Download Dropbox-Automation-Part2.fmw from the Files section of this article. Open it in FME Workbench and populate the DropboxConnector transformer with your Dropbox connection details. Note that this workspace is configured to watch the Dropbox path created in step 2 of the instructions for Part 1, and if a different directory structure is used, the AttributeCreator transformer just upstream of the DropboxConnector may need to be reconfigured for a shorter or longer file path.
When the workspace is configured, publish it to FME Flow under the Automations Exercises repository (create this if it doesn’t exist). Upload your Dropbox connection information with the workspace, and register it with the Job Submitter service.
3. Create a New Automation
From the FME Flow web interface, go to Automations > Create Automation to create a new Automation.
4. Configure the Trigger to Monitor a Dropbox
On the Automations canvas, double-click the Trigger node. Under Trigger, select Dropbox Directory (updated). Paste in an OAuth 2 token generated from your Dropbox app according to these instructions, which is the same OAuth 2 token generated in Part 1. Under Path to Watch, fill in the file path you created in Part 1, Step 2. Choose not to Watch Subdirectories (default), and remove DELETE from Events to Watch for.
Click Validate and (if valid) Apply. If the Trigger fails to validate, check the Dropbox file path you are watching and try re-pasting your Access Token, being careful not to highlight any leading or trailing whitespace.
5. Configure a Filter
Add an Action after the Dropbox Trigger, and select Filter messages from the list. Under Attribute, click the drop-down arrow, and select File Path. This reads the file path whenever the trigger detects a file |create or updated in Dropbox. Under Contains, enter .kml (do not use wildcards). We are monitoring this Dropbox for features digitized from Google Earth, and we will need to process them further to make them compatible with the rest of our project.
6. Configure a Run Workspace Action
Add an Action downstream of the success port on the Filter. Choose a Run Workspace Action, and select the Dropbox-Automation-Part2.fmw workspace you published to FME Flow earlier.
Populate Get the Automation’s upstream message by clicking the drop-down arrow, Event > Event as JSON. Event as JSON will pass all the information the automation has about the upstream workflow (in this case, just the Dropbox Directory Trigger) into the workspace. The workspace will parse the newly created Dropbox file path.
Click the ellipsis button under Folder for output GML and navigate to (or create) the Resource folder on FME Flow you wish to house output from this workspace. The workspace will automatically generate output file names based on the input name; simply specify the directory, e.g., $(FME_SHAREDRESOURCE_DATA)/Automations exercise data/Dropbox Exercises/GML from Dropbox/
7. Save and Start the Automation
Click Menu above the Automations canvas, then Save As. Choose a name and add some descriptive tags (optional), then click OK. Finally, click Start Automation in the upper right.
8. Test the Automation
In another browser tab, open the Dropbox web interface, navigate to the folder you are monitoring, and upload vancouverneighborhoods.kml from the Files section of this article (or another KML of your choice).
Wait a moment for the automation to trigger, then navigate back to your running automation. Select Menu > View Log File. You’ll see some lines related to the automation initializing and details about the Run Workspace job.
To jump right into the job log and omit the automation’s coordinating activities, instead choose Menu > View Triggered Jobs from the Automation canvas.
Conclusion
Nicely done! You are now prepared to automate workflows that upload files to a Dropbox folder or monitor and respond to changes in a Dropbox folder.
Troubleshooting
There are errors when creating a Web Connection through the DropboxConnector
If you receive an error when trying to connect your Dropbox account as a new connection via the DropboxConnector transformer, save and close FME Workbench, reopen it as an Administrator, then try creating the Web Connection again.
The Automation is failing with an invalid token
If your automation is occasionally failing due to an invalid token from Dropbox, this is due to Dropbox's OAuth 2.0 Authentication Service generating temporary tokens that last four hours before requiring a reset. There is currently no resolution for FME Flow; please see the Known Issues page for more information.
Data Attribution
The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.