Advanced API Calls

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2019.x

Tutorial: Tutorial: Getting Started with APIs | Previous: Download Files from a REST API Using FME  

Introduction

This article will continue to explore making API calls with FME. We previously worked through making API calls with the JSON Reader. This method is the simplest approach to working with APIs in FME, using a hardcoded request URL to retrieve air quality and pollution data from AirVisual’s REST API. Given that readers do not have input ports, we were unable to make a dynamic API call using attribute values from other datasets. To create API workflows that offer greater flexibility and functionality, there are several transformers that we recommend.

The following demonstration will provide instructions on how to access AirVisual’s API using three different transformers, all producing the same output KML file as the previous tutorial. The purpose of this demonstration is to compare the transformers that allow users to access APIs. They all offer distinct benefits and have optimal use cases depending on the requirements of the API and the desired output.

We will make API calls through the FeatureReader, the JSONExtractor, and the HTTPCaller. We will go through each approach from simplest to the most complex, describing the major differences and recommending use cases for each transformer. The usability of each method will depend on the response format and the requirements of the API. AirVisual’s API can be accessed from each of these transformers because:

  1. The response format is JSON
  2. The only request method AirVisual offers is GET
  3. There are no header or body parameters to fill out
  4. Authentication is done through a token attached to the request URL


Content Overview

Part 1: Create KML File
Part 2: API calls using a FeatureReader
Part 3: API calls using a JSONExtractor
Part 4: API calls using an HTTPCaller

Step-by-step Instructions

Part 1: Create KML File

Note: all complete workspaces will not run without an AirVisual API Key (see below).

To create the KML file, the first few steps for each of the three methods will be the same. If you have already retrieved an API key from AirVisual, you may continue with this demonstration. If not, follow steps 1 and 2 from the How to Access an API using a JSON Reader article to retrieve your own API key. If you would like to follow along using the coordinates of Safe Software’s head office, please use the following request URL:

http://api.airvisual.com/v2/nearest_city?lat=49.178&lon=-122.842&key=<YOUR API KEY>

1. Start FME Workbench

Select New workspace from the start-up page to open a blank canvas.


2. Add a Creator as a trigger for your workspace

The parameters can be left as default as this transformer is only used to initiate the workspace without having to input data. If you wanted to create a dynamic request URL, you could replace the Creator with an existing workflow containing valid GPS coordinates and an API key, as these are the only three elements of the request URL needed. For the purposes of this demonstration, we will be using a Creator to ensure that each method is comparable and consistent.


Part 2: API calls using a FeatureReader

The FeatureReader is a transformer that acts similarly to a reader but is incorporated into the workflow mid-translation. This means that there is an input port that requires initiation for the feature to be read. Like a typical reader, the FeatureReader can also read JSON, outputting values into an attribute with a specified coordinate system. The main difference between a FeatureReader and a reader is the presence of the input port, which allows for reading data mid-translation. The best use cases for this API approach would include situations where:

  • The API only offers one request method - GET
  • There are no header or body parameters
  • Every value in the response will be used
  • Dynamic request URLs are used
  • API authentication is required

featurereader.jpg

FeatureReader workflow

 

1. Add a FeatureReader to set up the call to AirVisual

Connect your Creator to the Initiator port of a FeatureReader. In the parameters, set the Format to JSON (JavaScript Object Notation) and the Coordinate System to LL84. This will ensure that our latitude and longitude values will be interpreted properly when we create a point feature. The Dataset will be the request URL with your API key substituted where <YOUR API KEY> appears.

featurereaderparams.jpg

Run the workspace with Feature Caching enabled to view your output in Visual Preview. Select the JSONFeature output port and open the Feature Information window in Visual Preview (Ctrl + Alt + F) to view a list of your attributes. Like the JSON Reader, every value from your response has been flattened into an attribute.

featurereaderattributes.jpg

The remaining steps to create geometry, manage attributes and write to KML are the same as steps 5 to 9 covered in the How to Access an API using a JSON Reader article. For a detailed description of each of the following steps, please refer to the linked article.

 

2. Add a VertexCreator to create a point feature from the coordinate array

Connect a VertexCreator to the JSONFeature output port from your FeatureReader. AirVisual returns the coordinates of the nearest monitoring station in a JSON array. Since these values are read into FME as list attributes, you will have to specify which value you intend to use by its index, which is indicated between the curly brackets. Note that JSON indexes begin at 0 and not 1.

coordinatearray.jpg

In the parameters of your VertexCreator, set the X Value as the first object in the coordinate array (Longitude), data.location.coordinates{0}, and the Y Value as the second (Latitude), data.location.coordinates{1}.

featurereadervertex.jpg

3. Refine the attributes to suit your needs

Add an AttributeManager and open the parameters. Double-click the cells under Output Attribute to rename your attributes. Rename the following attributes:

  • data.city ? City
  • data.state ? Province
  • data.country ? Country
  • data.current.weather.tp ? Temperature (C)
  • data.current.weather.hu ? Humidity (%)
  • data.current.weather.ws ? Wind Speed (m/s)
  • data.current.pollution.aqius ? AQI (US)
  • data.current.pollution.ts ? Date/Time

Use the black arrows below the table to arrange the order of your output attributes. To remove extra attributes, select their rows using Ctrl or Shift-click and then click the minus button below the table. Remove the remaining attributes.

attributemanagerfeaturereader.jpg


4. Add a Google KML Writer for your output

Select a file directory and name for your output .kml file. Change the Feature Type Definition to Automatic if it’s not already set and click OK. Rename the Feature Type Name to MonitoringStation_FeatureReader when prompted with the Feature Type window. Connect the AttributeManager Output port to the KML writer feature type.


5. Run the workspace and confirm that the .kml file exists

Click the green Run button in the toolbar and if the translation is successful, go to your file directory to make sure the .kml file was written.


6. Open your .kml file in Google Earth

In Google Earth, go to File > Open > Your .kml file

The view should automatically zoom to the location of your nearest monitoring station. Click on the pin to view your attribute data.


Part 3: API calls using a JSONExtractor

FME Desktop has a variety of JSON-specific transformers. Most of them allow the JSON input source to be a URL, meaning these tools are perfect for reading, parsing, and validating JSON data obtained from the web. The JSONExtractor in particular provides the best parsing capability, allowing users to easily create feature attributes out of the nested values buried in JSON format. Unlike the JSON Reader or the FeatureReader, the JSONExtractor will not automatically create attributes out of every value in the JSON text. Users have to specify their desired output attributes by selecting them from a JSON query. The benefit of using this transformer is that you will only output what you need from the API response. This is a handy feature that limits the amount of attribute handling later in the workspace and is especially useful when an API response returns many attribute values. The best use cases for this API approach would include situations where:

  • Only select attributes are needed from the API response
  • The API only offers one request method - GET
  • There are no header or body parameters
  • API Authentication is a token in the URL
  • The API response is JSON

jsonextractor.jpg

JSONExtractor workflow

 

1. Add a JSONExtractor to set-up the call to AirVisual

Connect your Creator to the JSONExtractor input port and open the parameters. Change the Input Source to File/URL and then copy your request URL into the File/URL parameter, substituting your AirVisual key where <YOUR API KEY> appears.

jsonextractorparam.jpg


2. Create feature attributes by extracting JSON queries

JSON queries (JavaScript property accessors in this case) describe the path to your attribute by fragmenting your data into its nested levels. If the value you need is deeply nested, your JSON query will be longer. Rather than manually typing in your query, we will select each one from the JSON tree.

Start by filling out the Target Attribute column in your JSONExtractor parameters. This includes the eight attribute names we’ve previously used, plus Latitude and Longitude.

  • City
  • Province
  • Country
  • Temperature (C)
  • Humidity (%)
  • Wind Speed (m/s)
  • AQI (US)
  • Date/Time
  • Longitude
  • Latitude

Next, select the blank space under the JSON Query column to expose the ellipses. Select the ellipses to view the JSON tree. Use the arrows to expand your JSON until you find the right attributes. For example, to find “city”, we would only need to expand “data”. When you click OK, the blank space will automatically populate with your JSON query.

jsontree.jpg

Some attributes are harder to identify because they are not self-explanatory like city, state, and country. For these abbreviated values, you will have to reference the API documentation to decode your response. Other attributes will be returned in an array, such as the coordinate values. To specify which array value you need, first select <array values> as your JSON query for both latitude and longitude. Once it has been populated under the JSON Query column, manually change the asterisk to 0 for longitude and 1 for latitude.

jsoncoordinatearray.jpg

jsonparameters.jpg


3. Add a VertexCreator to create a point feature

Unlike the JSON Reader and the FeatureReader, the Latitude and Longitude attributes can be directly inputted into the VertexCreator without having to worry about arrays because we have already separated the coordinates in the previous step. Set the X Value as your Longitude attribute and the Y Value as your Latitude.

vertex.jpg

4. Add a CoordinateSystemSetter to interpret your coordinates properly

Since there is no option to set a coordinate system in the JSONExtractor, we have to use a CoordinateSystemSetter to specify how the X and Y values will be read. Set the coordinate system to LL84.

css.jpg


5. Refine the attributes to suit your needs

Add an AttributeManager and open its parameters. You will notice that there are only the ten original attributes you extracted, plus the _creation_instance which is a product of the Creator. Remove the _creation_instance, Latitude, and Longitude attributes by using the drop-down menu that appears in the cells under the Action column.

attributemanagerjsonextractor.jpg


6. Add a Google KML Writer for your output

Select a file directory and name for your output .kml file. Change the Feature Type Definition to Automatic if it’s not already set and click OK. Rename the Feature Type Name to MonitoringStation_JSONExtractor when prompted with the Feature Type window. Connect the AttributeManager Output port to the KML writer feature type.


7. Run the workspace and confirm that the .kml file exists

Click the green Run button in the toolbar and if the translation is successful, go to your file directory to make sure the .kml file was written.


8. Open your .kml file in Google Earth

In Google Earth, go to File > Open > Your .kml file

The view should automatically zoom to the location of your nearest monitoring station. Click on the pin to view your attribute data.


Part 4: API calls using an HTTPCaller

The HTTPCaller is well-equipped to handle thousands of APIs without the need for coding. It accesses a request URL using a specified HTTP/HTTPS method, with the option to authenticate by username/password, named connection, headers, or uploaded bodies. With requests, users have the option to modify their query string parameters, headers, and if they were to PUT, POST, PATCH or DELETE, there is the option to upload a body. Responses can be saved to an attribute or file, while also providing error and status codes in the response header. The use cases for this API approach would include situations where:

  • The API requires more complex authentication methods such as Basic, Digest, or OAuth2
  • HTTP client options need to be modified, such as timeouts or saving cookies
  • The request URL is long or unreadable
  • HTTP methods beyond GET are called
  • The response is not JSON/XML
  • There are required headers

httpcaller.jpg

HTTPCaller workflow

 

1. Add an HTTPCaller to set-up the call to AirVisual

Connect your Creator to the input port and open the parameters. To improve readability, break up the components of your request URL using the Query String Parameters. Enter the following URL as your Request URL with the HTTP Method set to GET.

http://api.airvisual.com/v2/nearest_city?

The HTTPCaller lets you add URL query strings as transformer parameters. Enter lat, lon, and key under the Name column and then enter 49.178, -122.842, and your API key under the Value column where <YOUR API KEY> appears. The Query String Parameters is a great place to set up dynamic API calls because you can replace any cells in the Value column with an attribute or User Parameter. Dynamic calls with User Parameters is covered in the next article of this series. The remaining parameters can be left as default.

httpcallerparameters.jpg

Run the workspace with Feature Caching enabled to view your response body in Visual Preview. If you used the provided coordinates, the _response_body attribute should return something similar to this:

{"status":"success","data":{"city":"New Westminster","state":"British Columbia","country":"Canada","location":{"type":"Point","coordinates":[-122.91091999999999,49.20678]},"current":{"weather":{"ts":"2019-05-17T16:00:00.000Z","hu":71,"ic":"01d","pr":1008,"tp":12,"wd":160,"ws":3.6},"pollution":{"ts":"2019-05-17T15:00:00.000Z","aqius":12,"mainus":"p2","aqicn":24,"maincn":"n2"}}}}

 

2. Format your response to improve readability before extracting JSON queries

We have to manually create queries because the JSONExtractor does not display the JSON tree when reading a JSON document from an attribute. Therefore, we should format our JSON so we can read its structure to construct our queries.

We can format our JSON response with a JSONFormatter. Connect your HTTPCaller output to a JSONFormatter and set the JSON Document as your _response_body attribute. Leave the remaining parameters as default, and click OK. Run the workspace with Feature Caching enabled to view the reformatted JSON in Visual Preview. You should see something similar to this:

{
     "status" : "success",
     "data" : {
          "city" : "New Westminster",
          "state" : "British Columbia",
          "country" : "Canada",
          "location" : {
               "type" : "Point",
               "coordinates" : [ -122.91091999999999, 49.20678 ]
          },
          "current" : {
               "weather" : {
                    "ts" : "2019-05-17T16:00:00.000Z",
                    "hu" : 71,
                    "ic" : "01d",
                    "pr" : 1008,
                    "tp" : 12,
                    "wd" : 160,
                    "ws" : 3.6
               },
               "pollution" : {
                    "ts" : "2019-05-17T15:00:00.000Z",
                    "aqius" : 12,
                    "mainus" : "p2",
                    "aqicn" : 24,
                    "maincn" : "n2"
               }
          }
     }
}

The JSON text is now formatted with indentation and spaces to improve readability. The nested structure is clearly laid out, making it easier to construct JSON queries. For example, if we wanted to query the main pollutant according to the Air Quality Index of the United States, we could see that “mainus” is nested under “pollution” which is nested under “current” which is nested under “data”. Using this logic, we can create a JSON query:

json["data"]["current"]["pollution"]["mainus"]

To query values in an array, such as the coordinates of the monitoring station, you would use the same logic, but to specify an object in the array, a place number has to be indicated. For example, the JSON query for longitude is:

json["data"]["location"]["coordinates"][0]

We use zero to indicate longitude because it is the first value in the array. If we wanted to specify latitude, we would instead use 1 as the place number between the square brackets.


3. Extract your attributes from the JSON response body

In the JSONExtractor, set the JSON Document as _response_body or _formatted. These both contain the same JSON text, but one is formatted with indentation and spaces. Reformatting will not impact how JSON queries are executed.

Fill in the Target Attribute column using the name of each output attribute as follows:

  • City
  • Province
  • Country
  • Temperature (C)
  • Humidity (%)
  • Wind Speed (m/s)
  • AQI (US)
  • Date/Time
  • Longitude
  • Latitude

Under the JSON Query column, populate each cell with its corresponding query using the logic described in the previous step. It may be helpful to have the reformatted response body open in another window such as Notepad. Refer to the API documentation to decode the abbreviated values in the response body.

jsonextractorparams.jpg


4. Add a VertexCreator to create a point feature

The Latitude and Longitude attributes can be directly inputted into the VertexCreator without having to worry about arrays because we have already separated the coordinates in the previous step. Set the X Value as your Longitude attribute and the Y Value as your Latitude.

vertex.jpg


5. Add a CoordinateSystemSetter to interpret your coordinates properly

Since there is no option to set a coordinate system in the HTTPCaller, we have to use a CoordinateSystemSetter to specify how the X and Y values will be read. Set the coordinate system to LL84.

css.jpg


6. Refine the attributes to suit your needs

Add an AttributeManager and open its parameters. Remove _creation_instance, _response_body, _http_status_code, Longitude, Latitude, and _formatted by using the drop-down menu under the Action column to select Remove.

attributemanagerhttpcaller.jpg


7. Add a Google KML Writer for your output

Select a file directory and name for your output .kml file. Change the Feature Type Definition to Automatic if it’s not already set and click OK. Rename the Feature Type Name to MonitoringStation_HTTPCaller when prompted with the Feature Type window. Connect the AttributeManager Output port to the KML writer feature type.


8. Run the workspace and confirm that the .kml file exists

Click the green Run button and if the translation is successful, go to your file directory to make sure the .kml file was saved.


9. Open your .kml file in Google Earth

In Google Earth, go to File > Open > Your .kml file

The view should automatically zoom to the location of your nearest monitoring station. Click on the pin to view your attribute data.


Result

Each of the three methods described will produce the same .kml file, so if you used the provided coordinates, you should see something similar to this in all three cases:

googleearth.png

The output .kml file contains eight attributes that were parsed, renamed and reordered using different transformers in FME. In considering the final product, the most efficient method was the JSONExtractor because it allowed us to filter out select values, limiting the amount of attribute handling later in the workflow. Furthermore, the JSON tree allowed us to easily create queries, not requiring any previous knowledge of the JSON format. If we wanted every response value to be an output attribute, then the FeatureReader would be the best choice. Although the HTTPCaller method contained a JSONExtractor as well, we had to manually enter queries because the JSON tree is not an available feature when the input data source is dynamic. With this method, it is necessary to have a working knowledge of both APIs and JSON because there are more manual inputs.

The HTTPCaller offers the most user control in accessing an API, but for the purposes of this article, these features were not highlighted. For a demonstration of how to leverage the features of the HTTPCaller, visit the previous article in the API series “Download Files from a REST API Using FME”. You will learn how to access the Movebank REST API which returns a .csv file containing live animal movement data.

Additional Resources

Webinar: How to Connect to Any REST API (Without Writing Any Code)
Article: Getting Started with the Google Calendar API
Article: Getting Started with JSON

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.