Automating Dropbox Workflows with FME Flow

Introduction

This article is divided into two parts. The first will demonstrate how to upload data to Dropbox at the end of an FME Flow (formerly FME Server) automation workflow. The second will cover how to trigger an automation workflow when new data becomes available in Dropbox.

 

Requirements

• Dropbox account (create a free personal account)

 

Step-by-step Instructions

Part 1: Upload Files to Dropbox

In this workflow we will be creating an automation in FME Flow that will monitor a resource directory using the Resource and Network Directory (updated) Trigger and will copy incoming data into a Dropbox directory using the Dropbox Directory (upload) External Action.

We will also be configuring a Dropbox account with an app to generate an OAuth 2 access token, which will allow us to access the Dropbox directory.

1. Create a New Automation
From the FME Flow web interface, go to Automations > Create Automation to create a new automation.
 
2. Configure the Trigger to Monitor a Directory
On the Automations canvas, double-click the Trigger node. Under Trigger, select Resource or Network Directory.



Click the ellipsis button to the right of Directory to Watch and create the path Resources/Data/Automations/exercise data/Dropbox Exercises/toUpload.

It is recommended to create this exact path as it is referenced in the attached workspace used in Part 2 of this tutorial. 


 
Since we are monitoring for any incoming data and making a copy available in Dropbox, select Yes for both the Watch Subdirectories and Watch Folders parameters. Watch for CREATE and MODIFY events, but not DELETE. We want to make available in our Dropbox all of the most up-to-date data products, but don’t need to exactly mirror the contents of the monitored resource. For ease of testing the automation, set Poll Interval to 1 minute.


 
Click Validate and (if valid) Apply. If the Trigger fails to validate (a red bar containing a descriptive error message will appear), it is likely the path specified under Directory to Watch is incorrect; re-enter that parameter using the ellipsis (...) button.
 
3. Add an External Action
Downstream, configure a Dropbox Directory (upload to) External Action. If you are in guided mode, simply click on the silhouetted Next Action node and scroll down to External Actions. Otherwise, drag an External Action node out onto the canvas and connect it to the Resource or Network Directory  Trigger. Before we can proceed,  you will need to configure your Dropbox account with an app to generate an OAuth 2 access token. 

4. Configure Dropbox Account 
In Dropbox, create a new app called FME Flow Access that has full access.  



Once the app is created, navigate to the Permissions tab, select the following permissions, then click submit:

• files.metadata.write
• files.content.write
• files.content.read

If you leave your Dropbox app page, you can return any time by navigating back to developers.dropbox.com, clicking App console, and selecting the app.


 
Your OAuth 2 access token is separate from your access key and secret key. Select Settings after selecting the app, scroll down to OAuth 2, and click Generate under Generated Access Token. Copy the token. 




 
5. Configure Dropbox Directory External Action
Navigate back to the automation in FME Flow. In the Dropbox Directory External Action parameters, paste the token into the Access Token parameter.

Note: When permissions are changed, a new access token needs to be generated from the Dropbox application and then added to FME Flow, else the change will not be passed on to FME Flow.

Populate Source Path with any files created or modified in the watched directory by clicking the drop-down arrow and then selecting Directory > File Path. Choosing a directory will copy the contents of that directory and all subdirectories to your Dropbox.


 
Fill in Destination Path with a folder in your Dropbox where you wish to deposit the files. For example, if you want files copied to the root of your Dropbox, simply type “/”. If the folder you specify does not exist, it will be created the first time data is uploaded. Uploads will overwrite existing files with the same name in the Dropbox.

 

When you are satisfied with your configuration, click Validate and Apply.


 
6. Save and Start the Automation
Click Menu above the Automations canvas, then Save As. Choose a name and add some descriptive tags, then click OK. Click Start Automation in the upper-right.

 

7. Test the Automation
Navigate to the Resource folder your automation is monitoring (Resources/Data/Automations exercise data/Dropbox Exercises/toUpload is suggested above), then click Upload > Files and navigate to any file you wish to upload. Wait for at least one polling interval (1 min), then go to Automations > Manage, check the box next to the automation you just built, and select Actions > View Log File. As no workspaces were run by this automation, there will be no triggered jobs, but the Automation log file will include details of what files and file actions triggered a workflow, and where they were stored in the target Dropbox.


 
FME Flow logged the file upload, but we can always verify its success by logging into the target Dropbox and checking the folder path (/FMEFlow/watcherUploads/) we specified for the file uploaded to FME Flow earlier.

 

Well done! Your automation is now prepared to upload files to the Dropbox folder you specified whenever a file is created or modified in the folder FME Flow is monitoring. Next, we’ll monitor a folder in Dropbox, and trigger a workflow in FME Flow when files arrive in the Dropbox.
 

Part 2: Run a Workspace When Data Arrives in a Dropbox

In Part 2 of this exercise, we will imagine that users are uploading KML files of features digitized in Google Earth to a Dropbox, and that we want to monitor that Dropbox for new KML uploads, convert the KMLs to GML, and reproject them for use with other project data.
 
1. Create a Dropbox Folder to Monitor
In the Dropbox web interface, create a new folder to monitor for this exercise. Below, /FMEFlow/toWatch/ has been created.


 
2. Configure and Publish a Workspace to FME Flow
Download Dropbox-Automation-Part2.fmw from the Files section of this article. Open it in FME Workbench and populate the DropboxConnector transformer with your Dropbox connection details. Note that this workspace is configured to watch the Dropbox path created in step 2 of the instructions for Part 1, and if a different directory structure is used, the AttributeCreator transformer just upstream of the DropboxConnector may need to be re-configured for a shorter or longer file path.

When the workspace is configured, publish it to FME Flow under the Automations Exercises repository (create this if it doesn’t exist). Upload your Dropbox connection information with the workspace, and register it with the Job Submitter service.


 
3. Create a New Automation
From the FME Flow web interface, go to Automations > Create Automation to create a new Automation.
 
4. Configure the Trigger to Monitor a Dropbox
On the Automations canvas, double-click the Trigger node. Under Trigger, select Dropbox Directory (updated). Paste in an OAuth 2 token generated from your Dropbox app according to these instructions, which is the same OAuth 2 token generated in Part 1. Under Path to Watch, fill in the file path you created in Part 1, Step 2. Choose not to Watch Subdirectories (default), and remove DELETE from Events to Watch for.


 
Click Validate and (if valid) Apply. If the Trigger fails to validate, check the Dropbox file path you are watching and try re-pasting your Access Token, being careful not to highlight any leading or trailing whitespace.
 
5. Configure a Filter
Add an Action after the Dropbox Trigger, and select Filter messages from the list. Under Attribute, click the drop-down arrow, and select File Path. This will read in the file path whenever the trigger registers that a file is created or updated in Dropbox. Under Contains, enter .kml (do not use wildcards). We are monitoring this Dropbox for features digitized from Google Earth, and we will need to do some further processing on them to make them compatible with the rest of our project.

 
6. Configure a Run Workspace Action
Add an Action downstream of the success port on the Filter. Choose a Run Workspace Action, and select the Dropbox-Automation-Part2.fmw workspace you published to FME Flow earlier.

Populate Get the Automation’s upstream message by clicking the drop-down arrow, Event > Event as JSON. Event as JSON will pass all the information the automation has about the upstream workflow (in this case, just the Dropbox Directory Trigger) into the workspace. The workspace itself will parse out the newly-created Dropbox file path.

Click the ellipsis button under Folder for output GML and navigate to (or create) the Resource folder on FME Flow you wish to house output from this workspace. The workspace will automatically generate output file names based on the input name; simply specify the directory, e.g. $(FME_SHAREDRESOURCE_DATA)/Automations exercise data/Dropbox Exercises/GML from Dropbox/

 

7. Save and Start the Automation
Click Menu above the Automations canvas, then Save As. Choose a name and add some descriptive tags (optional), then click OK. Finally, click Start Automation in the upper right.


 
8. Test the Automation
In another browser tab, open the Dropbox web interface, navigate to the folder you are monitoring, and upload vancouverneighborhoods.kml from the Files section of this article (or another KML of your choice).


 
Wait a moment for the automation to trigger, then navigate back to your running automation. Select Menu > View Log File. You’ll see some lines related to the automation initializing and details about the Run Workspace job.


 
To jump right into the job log and omit the automation’s coordinating activities, instead choose Menu > View Triggered Jobs from the Automation canvas.
 

Conclusion

Nicely done! You are now prepared to automate workflows that upload files to a Dropbox folder or monitor and respond to changes in a Dropbox folder.

 

Troubleshooting

There are errors when creating a Web Connection through the DropboxConnector

If you receive an error when trying to connect your dropbox account as a new connection via the DropboxConnector transformer, save and close your FME Workbench and re-open it as an Administrator, then attempt to make the Web Connection again.

 

The Automation is failing with an invalid token

If your automation is occasionally failing due to an invalid token from Dropbox, this is due to Dropbox's OAuth 2.0 Authentication Service generating temporary tokens which last four hours before requiring a reset. There is currently no resolution for FME Flow, please see the Known Issues page for more information on FME Form.

 

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.