Authenticating and Extracting Information from API calls with the HTTPCaller

Introduction 

FME can interact with APIs through a variety of readers, writers, and transformers without the need to code. For web service providers that are not natively supported by FME, the HTTPCaller transformer can be used to access a sea of APIs. This tutorial will walk you through how to “wrap up” an API into a custom transformer. By creating a custom transformer you are able to take a sequence of transformers needed to access the information in an API into a single object on the canvas helping to tidy up your workspace as well as make your transformation sequence shareable by publishing to the FME Hub.  
 

Precisely’s APIs

In this tutorial, we will access one of Precisely’s APIs as an example on how to create a custom transformer that outputs data that is more digestible which you could use to enrich your data.
Precisely provides services to access 100+ APIs that work to enrich your data and power your applications. Learn more about Precisely’s APIs here.
For this tutorial, we will create a custom transformer using Precisely’s Address Verification API. The Address Verification API works to eliminate errors in address data, improving customer experience, keeping it up-to-date and maintaining its accuracy and consistency. 

 

Step-by-step Instructions

Part 1: Generate a Token 

The documentation of the API you are looking to utilize will likely mention the method required to authorize your first API call. Precisely’s APIs require you to generate an OAuth token by encoding your API Key and Secret. Let’s set this OAuth token up in FME
Start by putting a creator down on your canvas and connect the output port to an HTTPCaller. 
Configure the HTTPCaller according to how your API’s documentation requires it. In this example, the HTTP Method is set to POST and the Header and Body has been configured to carry the Authorization Key. The output of HTTPCaller will be stored in the _response_body

Configure the HTTPCaller according to how your API’s documentation requires it. In this example, the HTTP Method is set to POST and the Header and Body has been configured to carry the Authorization Key. The output of HTTPCaller will be stored in the _response_body. 

Upon running the HTTPCaller we get a _reponse_body with the access token embedded in the JSON body. 

Next, add some data to test the first API Call. This tutorial will use mock census data. Use a FeatureMerger to add the _response_body to each of the attributes in the census data. Merge according to the creation instance. 

Connect the output of the FeatureMerger to a JSONFlattener. The JSONFlattener will extract the object keys and values into FME feature attributes. Configure the JSONFlattener to extract “access_token”. 
 

In the output, we see that all features now have an access_token attribute. We are now ready to use these features in our first API Call. 
 

Part 2: POST HTTP Method: Configure the HTTPCaller to make your first API Call

Put down another HTTPCaller onto the Workspace. Refer back to the documentation for the API for the request parameters of the API. 
For the Request URL: Input https://api.precisely.com/addressverification/v1/validatemailingaddress/results.json as we want our results in JSON. 
For the HTTP Method select POST as outlined in the API documentation. 
Configure the Header to match the API documentation. 
Configure the Upload Body according to the sample request body. 
Set Content Type as JSON.  

Confirm your configurations and run your HTTPCaller. 
Inspect your _reponse_body. This holds all the valuable information we need. All there is left to do is to parse this and make the results digestible. 
 

Part 3: Parsing the response body

These transformers come in handy when parsing JSON

• JSONFragmentor
• JSONFlattener
• JSONExtractor
• ListExploder

Let's create a sequence of these transformers that will correctly extract information from the response body of our API every time. You can visualize your response body in any JSON Parser of your choice to understand the schema of your API call. 
 

​​​​​​​First, put down a JSONFragmenter to fragment a section of your response body you want to flatten. Select your _reponse_body to be your JSON Attribute. Input json["Output"][*] into your JSON Query.
 Connect the output to your JSONFlattener For Attributes to Expose there are a few ways you can go about doing this. 

• Manual Selection
• Import Schema 

To manually select attributes to flatten, enter the values into the “Attributes to Expose”. To Import Schema from a source select Import to bring in a source document that follows the same schema. You could achieve this by simply saving a .json file of your _response_body. 

​​​​​​​Run the JSONFlattener and your selected attributes will be exposed in your table. 
 

Part 4: Create a Custom Transformer

The sequence of transformers needed to extract information from our API has now been completed. Now it is time to tidy up this series inside a custom transformer. Select all relevant transformers in your canvas. Right-click or (Ctrl + T) to create your custom transformer. Insert transformer Inputs and Outputs to best fit your needs. Refer to the Complete Guide to FME Custom Transformers for best practices.

• Link to The Complete Guide to FME Custom Transformers

​​​​​​​

​​​​​​​Optional: Publish to the FME Hub

Share your custom transformer to the FME Hub to share your creation with others looking to achieve similar workflows. 

•  Link to FME Hub Best Practices