Download Files from a REST API Using FME

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2022.0

Introduction

This article will provide instructions on how to access Movebank’s REST API using the HTTPCaller. Movebank is a public database containing animal movement data from scientific studies all over the world.

In this demonstration, the HTTPCaller returns data on the movement of a bald eagle from a study based in British Columbia. The workspace will allow users to choose one of the ten tagged birds from the study at run time. The HTTPCaller will receive a response from the Movebank API containing spatial and temporal data about the selected eagle. Using the timestamp of each GPS reading, we will categorize the point locations by season to create a GeoJSON that displays migratory patterns of the eagle throughout the year.

Movebank’s API requires Basic authentication which can be done using three different methods in FME:

  • The Authentication option in HTTPCaller - Check the box to Use Authentication and specify Basic as your Authentication Method. Fill in your Movebank username and password directly into the HTTPCaller. This method will be used in the following demonstration.
  • The Headers parameter in HTTPCaller - use the Authorization header and encode your Movebank username and password in Base64 using this format username:password
  • Web Connection - Create an HTTP Authentication Service from Tools → FME Options → Web Connections. Once it is authenticated, you should be able to select it from the HTTPCaller. This method allows you to securely share credentials using FME Server.

Although the Movebank API offers request URLs that return JSON data, the request URL that we are interested in only returns CSV. Since we are modifying the request URL with each run, the Query String Parameters in the HTTPCaller will be used to improve readability and incorporate User Parameters. Because we are making dynamic calls to an API that requires Basic authentication and returns CSV data, Movebank is the ideal API to handle in the HTTPCaller.
 

Step-by-step Instructions

1. Create a Movebank Account to Gain Permissions for Animal Tracking Data

Since Movebank data is managed by the owners of the respective study, permissions for data access can vary depending on the dataset needed. While some studies require users to read and accept their terms of use, the data we are interested in only requires a username and password.

To get started, create a Movebank account. Select Data > Map on the header, and then select Login on the Map's banner menu. 


On the Login popup window, select Register. Once you sign up, check the email that is immediately sent containing an activation link. After activating your account, return to the Movebank Map. 
 

2. View the Metadata for the Bald Eagle Study

Using the Search bar, type in “Bald Eagles” and zoom into western Canada. You should see one green point near Vancouver, BC. Click on the point to view additional information and select “Open in Studies Page”.


The Study Details outline all the relevant information provided by both Movebank and the owner of the data. Note the Movebank ID number which is a unique key for accessing the data in this study. We will use it later as a parameter in our request URL. 


3. View Movebank’s API Documentation

Movebank’s API Documentation is shared on GitHub where additional security, data access and authentication information is also detailed. Since we are creating a workflow that allows users to select a specific eagle at run time, select “Get information about animals in a study” from the Contents.

documentation1.png


Note the request URL that we will be using to obtain a list of unique eagle IDs will take the form:

https://www.movebank.org/movebank/service/direct-read?entity_type=individual&study_id=2911040

Referring back to the Contents, now select “Get event data for an individual animal”.

documentation.png


Note the request URL that we will be using to obtain tracking data for our selected eagle will take the form:

https://www.movebank.org/movebank/service/direct-read?entity_type=event&study_id=2911040&individual_id=2911059


4. Start FME Workbench

Select New workspace from the start-up page to open a blank canvas.


5. Add a Creator as a Trigger for the Workspace

Add a Creator to the canvas. The parameters can be left as default as this transformer is only used to initiate the workspace without having to input data.


6. Add an HTTPCaller to Set Up the First Call to Movebank

Connect your Creator to the input port of an HTTPCaller and open the parameters. Our first call to the Movebank API will use the request URL to get information about animals in a study. Use the following request URL with the HTTP Method set to GET:

https://www.movebank.org/movebank/service/direct-read?entity_type=individual&study_id=430263960

Change where the response body is saved to by selecting File from the Save Response Body To drop-down menu. For the Output Filename, select the ellipses to choose a directory, making sure that you add “.csv” as a file extension onto the name of your output.

saveoutput.jpg

Check the box to Use Authentication. Select Basic as your Authentication Method and enter your Movebank account's username and password.

HttpCaller.png

 

7. Run the Workspace to Ensure that the CSV File is Saved in the Correct Directory

Click the green Run button. When the translation is completed successfully, check your directory to make sure the file was saved. Open the file in a text editor or Microsoft Excel to view the data. The fifth column will show the individual bird IDs that we will be entering as a user parameter.

birdid.jpg

 

8. Create a User Parameter to Select an Individual Eagle at Run Time

Right-click User Parameters in the Navigation pane and select Manage User Parameters. In the User Parameter manager, click on the green plus sign (+) in the top-left corner, and then select Choice. This parameter will provide the user with a drop-down list of Bird IDs at run time.
SelectChoice.png

On the right-hand side of the parameter manager, configure the parameter as follows:

  • Parameter Identifier: individual_id
  • Prompt: Individual Bird ID
  • Published: Enabled
  • Required: Enabled
  • Expand Choice Configuration:
    • Dropdown: Enabled
Next, we'll import the values. Click on the Import button then select from Dataset. The Import Wizard will open prompting you for a Format and Dataset. Set the Format as CSV and choose your CSV file from the previous step as the Dataset.

Import1.png


Click Next. Change the Import from to Attribute values.

14.png


Click Next and select id as both the Value and Display. Click Import. 
Import3.png

 

A list of 12 individual eagle IDs will appear. Ensure the Default Value is blank, then click OK. 
PublishedParam.png

 

9. Update HTTPCaller

We will reuse the HTTPCaller from before since we are done with this request. The next request URL will receive tracking data for an individual from our eagle study:

https://www.movebank.org/movebank/service/direct-read?entity_type=event&study_id=430263960&individual_id=<INDIVIDUAL EAGLE ID>

We will use the Query String Parameters to break up the components of the URL. Enter the following URL into the request URL field and set the method to GET:

https://www.movebank.org/movebank/service/direct-read?

Enter entity_type, study_id, and individual_id under the Name column of the Query String Parameters as three different rows. In the Value column, set the entity type as event, and the study ID to 430263960. For the individual ID, click on the arrow that appears in the Value column when you click inside the cell. Move your cursor down to User Parameter and then select individual_id.

Change the Output Filename to a directory of your choice renaming the file to TrackingData.csv.
Http2.png

 

10. Run the workspace to make sure the HTTPCaller is working

Make sure your User Parameter will be prompted every time you run the workspace by clicking on Run along the top toolbar and then Prompt for User Parameters. A checkmark should appear next to this text. Run the workspace and you will be prompted to select an Individual Bird ID. Select the first value from the list 430270991 and click Run. Check your directory to make sure the response body was saved, and that it contains data.

 

11. Add a FeatureReader to read in your CSV file

Connect your HTTPCaller output to a FeatureReader. In the parameters, set the Format as CSV and use the arrow to the right of Dataset to specify the _response_file_path attribute as your input. Set the coordinate system to LL84 and then expand Output Ports to select Single Output Port.

FeautureReader.png
 

Run the workspace with Feature Caching enabled to view the output in Visual Preview. You can choose any ID when prompted. The Generic port will have hundreds to thousands of output features but if you click on the magnifying glass next to <Generic>, you will only see empty rows in Visual Preview. This is because the attributes are unexposed. You can still see them by opening the Feature Information window (Ctrl + Alt + F) and selecting a row from the table. Scroll through this window and you will see our five attributes.

hidden.jpg

 

12. Add an AttributeExposer to reveal hidden attributes

Dynamic source data causes attributes to be hidden from a workspace so we have to expose them in order to use the attributes downstream. Connect the Generic output port of the FeatureReader to an AttributeExposer. Fill in the Attributes to Expose using the five attribute names from the Feature Information window in the previous step (you could also use Import on TrackingData.csv and choose Attribute Names instead of Values, if you prefer).

AttributeExposer.png

Run the workspace again and you will see that the attributes are now visible in Visual Preview.

 

13. Add a DateTimeConverter to reformat the timestamp attribute

To categorize our data by season, we have to first format the timestamp attribute to only display month and day. Connect the AttributeExposer to the DateTimeConverter and open the parameters. Select the timestamp attribute from the drop-down menu next to the Datetime Attributes field. For Output Format, enter %m%d. Click OK.

DateTimeConverter.png

 

14. Add an AttributeRangeMapper to categorize dates into seasons

The AttributeRangeMapper will create a new attribute based on numeric ranges. Select timestamp as the Source Attribute and set the Output Attribute as Season. Since equinoxes and solstices vary by year, we will go by the 2022 calendar year to indicate start and end dates of each season. In the first row of the Range Lookup Table, enter 0000 in the From column and 0319 in the To column. Set the Output Value as Winter. This indicates that from the beginning of the year (which would be 0101) to March 19th, the season is Winter. Repeat this process for the following ranges:

  • From 0320 To 0620 → Spring
  • From 0621 To 0922 → Summer
  • From 0923 To 1220 → Fall
  • From 1221 To 1231 → Winter

RangeMapper.png

 

15. Add a GeoJSON writer for your output

Add a Writer and specify the Format as GeoJSON (Geographic JavaScript Object Notation). Write a file named EagleOutput.json to your desired output folder. Ensure Feature Type Definition is set to Automatic. Click OK and you will be prompted to change the Feature Type Name - enter BaldEagle. Connect your AttributeRangeMapper to the Writer. If you look at the Published Parameters in your Navigation pane, you will notice a new one has appeared which asks users to specify a destination for their output. Because every run may contain different data, it is important that users be able to specify a different file name to avoid overwriting their data with every run.

 

16. Run the workspace and ensure that the GeoJSON file is saved in the correct directory

You will be prompted to select a bird and a directory for your GeoJSON when you click the green Run button. Fill in the parameters and click Run. Once the translation is successful, go to the directory to make sure the file was written.

Prompt.png

 

17. View the migratory patterns of your select bald eagle

To view migratory patterns, you will have to symbolize your data by the Season attribute. There are many applications that will directly read a GeoJSON file (i.e. QGIS, ArcMap). In this example, we will be using ArcGIS Online (you can also do a quick visualization by reading in the GeoJSON with FME Workbench and then sending it to an Inspector transformer with a Group-By set on Season). Upload your GeoJSON file to ArcGIS Online as a hosted layer and open it in the Map Viewer. When prompted to choose an attribute to show, select Season. The output map will vary depending on which individual ID you selected, but migratory patterns should remain similar. In the Spring and Summer, bald eagles along the coast of BC tend to migrate North towards the Gulf Islands, whereas they fly South in the Fall and Winter when temperatures drop.

ago.png


Data Attribution

The data used in this article are available on Movebank (movebank.org, study name "Bald Eagle (Haliaeetus leucocephalus) in British Columbia, ID=430263960") and are published in the Movebank Data Repository. 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.