Download Files from a REST API Using FME

Introduction

This article provides instructions on accessing Movebank’s REST API using the HTTPCaller. Movebank is a public database that contains animal movement data from scientific studies worldwide.

In this demonstration, the HTTPCaller returns data on the movement of a bald eagle from a study based in British Columbia. The workspace will allow users to select one of the tagged birds from the study at runtime. The HTTPCaller will receive a response from the Movebank API containing spatial and temporal data about the selected eagle. Using the timestamp of each GPS reading, we will categorize the point locations by season to create a GeoJSON that displays migratory patterns of the eagle throughout the year.

Movebank’s API requires Basic authentication, which can be done using three different methods in FME:

• For the authentication option in HTTPCaller, check the box to use authentication and specify basic as your authentication method. Then, fill in your Movebank username and password directly into HTTPCaller. This method will be used in the following demonstration.
• The Headers parameter in HTTPCaller - use the Authorization header and encode your Movebank username and password in Base64 using this format username:password
• Web Connection - Create an HTTP Authentication Service from Tools → FME Options → Web Connections. Once it's authenticated, you should be able to select it from the HTTPCaller. This method allows you to securely share credentials using FME Flow.

Although the Movebank API offers request URLs that return JSON data, the request URL that we are interested in only returns CSV. Since we are modifying the request URL with each run, the HTTPCaller's Query String Parameters will be used to improve readability and incorporate User Parameters. Because we are making dynamic calls to an API that requires Basic authentication and returns CSV data, Movebank is the ideal API to handle in the HTTPCaller.

Step-by-step Instructions

1. Create a Movebank Account to Gain Permissions for Animal Tracking Data

Since Movebank data is managed by the owners of the respective study, permissions for data access can vary depending on the dataset needed. While some studies require users to read and accept their terms of use, the data we are interested in only requires a username and password.

To get started, create a Movebank account. Select Data > "Explore map" on the header, and then select Login on the Map's banner menu. 

On the Login pop-up window, select Register. Once you sign up, check your email for an activation link. After activating your account, return to the Movebank Map. 

2. View the Metadata for the Bald Eagle Study

Using the Search bar, type in “Bald Eagles” and zoom into Western Canada. You should see one point near Vancouver, BC. Click on the point to view additional information and select “Open in Studies Page”.

The Study Details outline all the relevant information provided by both Movebank and the data owner. Note the Movebank ID number, which is a unique key for accessing the data in this study. We will use it later as a parameter in our request URL. 

3. View Movebank’s API Documentation

Movebank’s API Documentation is shared on GitHub, where additional security, data access, and authentication information is also detailed. Since we are creating a workflow that allows users to select a specific eagle at run time, select “Get information about animals in a study” from the Contents.

Note that the request URL that we will be using to obtain a list of unique eagle IDs will take the form: https://www.movebank.org/movebank/service/direct-read?entity_type=individual&study_id=2911040

Referring back to the Contents, select “Get event data for an individual animal” now.

Note the request URL that we will be using to obtain tracking data for our selected eagle will take the form:

https://www.movebank.org/movebank/service/direct-read?entity_type=event&study_id=2911040&individual_id=2911059

4. Start FME Workbench

Select New Workspace from the start-up page to open a blank canvas.

• Add an HTTPCaller to the canvas and double-click on it to open the parameters

• Add a Creator to the canvas. The parameters can be left at their defaults, as this transformer is only used to initiate the workspace without requiring data input.

  Connect your Creator to the input port of an HTTPCaller and open the parameters.

5. Configure HTTPCaller to Set Up the First Call to Movebank

Our first call to the Movebank API will use the request URL to get information about animals in a study. Use the following request URL with the HTTP Method set to GET:

https://www.movebank.org/movebank/service/direct-read?entity_type=individual&study_id=430263960

Change where the response body is saved by selecting File from the "Save Response Body To" drop-down menu. For the Output Filename, select the ellipses to choose a directory, making sure that you add “.csv” as a file extension to the name of your output.

Check the box to Use Authentication. Select "Basic" as your Authentication Method, then enter your Movebank account username and password.

6. Run the Workspace to Ensure that the CSV File is Saved in the Correct Directory

Click the green Run button. Once the translation is complete, check your directory to confirm the file has been saved. Open the file in a text editor or Microsoft Excel to view the data. The "id" column shows the individual bird IDs we will enter as a user parameter.

7. Create a User Parameter to Select an Individual Eagle at Run Time

Right-click "User Parameters" in the Navigation pane and select "Manage User Parameters". In the User Parameter manager, click on the green plus sign (+) in the top-left corner, and then select "Choice". This parameter provides the user with a drop-down list of Bird IDs at runtime.

On the right-hand side of the parameter manager, configure the parameter as follows:

• Parameter Identifier: individual_id
• Prompt: Individual Bird ID
• Published: Enabled
• Required: Enabled
• Expand Choice Configuration:
  • Dropdown: Enabled

Next, we'll import the values. Click on the Import button, then select "From Dataset". The Import Wizard will open, prompting you for a Format and a Dataset. 

• Format: CSV (Comma Separated Value)
• Dataset: CSV file from the previous step

Click Next. Change the "Import from" to "Attribute values".

Click Next, then select "id" for both Value and Display. Click Import. 

A list of individual eagle IDs will appear. Ensure the Default Value is blank, then click OK.

8. Update HTTPCaller

We will reuse the HTTPCaller from before since we are done with this request. The next request URL will receive tracking data for an individual from our Eagle study:

https://www.movebank.org/movebank/service/direct-read?entity_type=event&study_id=430263960&individual_id=<INDIVIDUAL EAGLE ID>

We will use query string parameters to split the URL components. Enter the following URL into the request URL field and set the method to GET:

https://www.movebank.org/movebank/service/direct-read?

Set the query string parameters as follows:

• Query String Parameters:
  • Name: entity_type
    • Value: event
  • Name: study_id
    • Value: 430263960
  • Name: individual_id 
    • Value: individual_id
      • Click on the arrow that appears in the Value column when you click inside the cell. Move your cursor down to User Parameter and then select individual_id

Change the Output Filename to a directory of your choice and rename the file to "TrackingData.csv".

9. Run the Workspace

Ensure that your User Parameter is prompted every time you run the workspace by clicking "Run" in the top toolbar, then selecting "Prompt for Parameters". A checkmark should appear next to this text. Run the workspace, and you will be prompted to select an Individual Bird ID. Select the first value from the list and click Run. Check your directory to ensure the response body was saved and contains data.

10. Add a FeatureReader to Read in the CSV

Run the workspace with Feature Caching enabled to view the output in Visual Preview. You can choose any ID when prompted. The Generic port will have hundreds to thousands of output features, but if you click on the magnifying glass next to <Generic>, you will only see empty rows in Visual Preview. This is because the attributes are unexposed. You can still see them by opening the Feature Information window (Ctrl + Alt + F) and selecting a row from the table. Scroll through this window to view our five key attributes.

11. Add an AttributeExposer to Reveal Hidden Attributes

Dynamic source data causes attributes to be hidden from a workspace; therefore, we must expose them to use them downstream. Connect the Generic output port of the FeatureReader to an AttributeExposer. Fill in the "Attributes to Expose" using the five attribute names from the Feature Information window in the previous step (you could also use Import on TrackingData.csv and choose Attribute Names instead of Values, if you prefer).

Run the workspace again, and the attributes will be visible in Visual Preview.

12. Add a DateTimeConverter to Reformat the Timestamp Attribute

To categorize our data by season, we first have to format the timestamp attribute to display only month and day. Connect the AttributeExposer to the DateTimeConverter and open the parameters. Select the "timestamp" attribute from the drop-down menu next to the Datetime Attributes field. For Output Format, enter "%m%d". Click OK.

13. Add an AttributeRangeMapper to Categorize Dates into Seasons

The AttributeRangeMapper will create a new attribute based on numeric ranges. Select "timestamp" as the Source Attribute and set the Output Attribute as "Season". Since equinoxes and solstices vary by year, we will go by the 2022 calendar year to indicate the start and end dates of each season. In the first row of the Range Lookup Table, enter 0000 in the From column and 0319 in the To column. Set the Output Value as Winter. This indicates that from the beginning of the year (which would be 0101) to March 19th, the season is Winter. Repeat this process for the following ranges:

• From 0320 To 0620 → Spring
• From 0621 To 0922 → Summer
• From 0923 To 1220 → Fall
• From 1221 To 1231 → Winter

14. Add a GeoJSON Writer

Add a writer and specify the Format as GeoJSON (Geographic JavaScript Object Notation). Write a file named "EagleOutput.json" to your desired output folder. Ensure the Feature Type Definition is set to "Automatic". Click OK, and you will be prompted to change the Feature Type Name - enter "BaldEagle". Connect your AttributeRangeMapper to the Writer. If you look at the Published Parameters in your Navigation pane, you will notice a new parameter that asks users to specify a destination for their output. Because each run may contain different data, it's important that users can specify different file names to prevent their data from being overwritten in subsequent runs.

15. Run the Workspace

You will be prompted to select a bird and a directory for your GeoJSON when you click the green Run button. Fill in the parameters and click Run. Once the translation is successful, go to the directory to make sure the file was written.

16. View the Migratory Patterns of your Select Bald Eagle

To view migratory patterns, you will have to symbolize your data by the Season attribute. There are many applications that can read GeoJSON files directly (e.g., QGIS, ArcMap). In this example, we will use ArcGIS Online (you can also do a quick visualization by reading in the GeoJSON with FME Workbench and then sending it to an Inspector transformer with a Group-By set to Season). Upload your GeoJSON file to ArcGIS Online as a hosted layer and open it in the Map Viewer. When prompted to choose an attribute to display, select "Season." (If the prompt does not show up, you can navigate to Styles on the right-side menu, click on Field, select your "Season" attribute, and choose "Types (unique symbols)" as your Style.)

The output map will vary depending on which individual ID you selected, but migratory patterns should remain similar. In the spring and summer, bald eagles along the coast of BC tend to migrate north towards the Gulf Islands, whereas they fly south in the fall and winter when temperatures drop.

Data Attribution

The data used in this article are available on Movebank (movebank.org, study name "Bald Eagle (Haliaeetus leucocephalus) in British Columbia, ID=430263960") and are published in the Movebank Data Repository.