Making a GET Request with HTTPCaller

Introduction

The HTTPCaller transformer allows you to access a URL from within your FME Workbench workflow. HTTPCaller has built-in parameters to handle all of your API request needs, see the full list of options in the transformer documentation.
 

This example will demonstrate a basic HTTP GET request to the Cat Facts API. We will then parse the API’s JSON response and send an email with its content. The end result is a workspace that uses an API to send a random fact about cats to a recipient of your choice.

 

Step-by-Step Instructions

1. Create a Workspace

Open FME Workbench and start a New workspace to begin building your workflow on an empty canvas. 

2. Start the Workflow 

Because we will be getting our data from an API, we don’t need to use a reader for incoming files or connections. The Creator transformer lets us start a workflow by creating new features that we can process later in the workspace. Add the Creator transformer by clicking anywhere on the canvas and typing “Creator”.

3. Add an HTTPCaller

Next, add an HTTPCaller to the canvas and connect the Creator to it. 

Open the HTTPCaller and take a look around. Note that there are two required fields: Request URL and HTTP Method. We will need at least those two pieces of information to make a call. There are many other parameters we can optionally fill out. For now, select Cancel to leave the parameters empty.

4. Review the API Documentation

Before we can fill out the HTTPCaller parameters, we should consult the API’s documentation. For this example, we will use Cat Facts API. Cat Facts is a simple API designed to do one thing: send facts about cats. Scroll down to its README.md to learn more about its functionality. 

Tip: When choosing an API, always review the Terms and Conditions to ensure it fits your needs. 

Select Start Developing! under API Documentation. 

The Base URL is the main access point to the API, and the root of all your Request URLs: 

https://cat-fact.herokuapp.com

Endpoints allow you to access and exchange specific information made available by the API. Specify the endpoint by appending it to the end of the base URL. Cat Facts has two endpoints: /user and /facts. The two Models, Fact and User, refer to data objects accessible through the API. 

5. Start Building the API Call

Our goal is to retrieve a cat fact from the API and bring it into the workspace. With this in mind, we can start building our API Call. Return to the workspace and open up the HTTPCaller parameters. 
 

In the Request URL, enter the base URL from the documentation with the endpoint we want to reach.

https://cat-fact.herokuapp.com/facts

Tip: If you were to enter this URL in your browser, the JSON data model is returned. Nested inside key value pairs, you can preview some of the Cat Facts we’re hoping to bring into our workflow. If you are new to working with JSON, see the Getting Started with JSON tutorial to learn more about working with JSON in FME.

Since we just want to retrieve data from the API, we will use a GET request. Set the HTTP Method to GET. For other calls you might make, the HTTPCaller can handle many more request types. 
 

You have the option to write the API response to an Attribute or File. For this example, we’d like to save the response body to an Attribute _response_body so that we can use it downstream. 

Though there are many other optional parameters available on the HTTPCaller, the Request URL and HTTP Method are all we need to modify for this request. 

It is notable that the Cat Facts API does not require authentication for the /users endpoint. Thus, we do not need to select the Use Authentication parameter. But for many other APIs, there is typically some type of authentication for access.
 

Select OK. 

6. Test the API Call

Turn on Feature Caching and select Run. 

Examine the output by selecting the looking glass icon on the HTTPCaller transformer. A successful translation returns a 200 value to the _http_status_code attribute. This means that the request was received OK and the message has been fetched from the API server. There are many response codes you may receive, it is good practice to familiarize yourself with them. 

Note: If the workspace fails, the API may be down. Either retry in a couple of minutes or in the HTTPCaller replace the Request URL with:

http://fme.ly/CatFact

7. Examine the API Response

From the table, we can double-click the _response_body attribute to inspect what the API has returned. 

This isn’t the easiest to read, but we can deduce that it is nested JSON key-value pairs. You can format the JSON to make it easier to read directly inside the workspace. Add a JSONFormatter to your canvas and connect it to the HTTPCaller. 

Open up the JSONFormatter and set the JSON Document parameter to the Attribute Value, _response_body. Ensure the Formatting Type is set to Pretty Print. Run the workspace to the JSONFormatter. 


 

Examine the output and double-click on the new attribute, _formatted. The key-value pairs are now much easier to read. With some understanding of JSON, we can now choose what data we would like to bring into our workflow. The facts about cats are string values for “text”, nested inside the entire array. We need to isolate those values to use them in our workflow.

8. Parse the JSON Body

A JSONFragmenter extracts JSON portions into new FME features. Add a JSONFragmenter transformer to the workspace and connect it to the HTTPCaller’s output port. 

Open it up to examine its parameters. The JSON Attribute is the JSON body we want to target, set it to the _response_body attribute. In the JSON Query, we specify what values we want to become new features. Previously, we found the cat facts were associated with the “text” key, which was part of the entire array. To isolate this value we come up with the following query: 

json[*]["text"]

This finds the “text” values in all the objects in the entire array. If you’d like to learn more about building JSON queries, please refer to the JSONFragmenter documentation. 
 

Enter the query in the JSON Query parameter. The rest of the parameters are useful for creating attributes from more complex JSON, but can be left default for our purposes. 

9. Test the JSON Query

Ensure we’ve built the query correctly by running the workspace to the JSONFragmenter. Inspect the features from the Fragments port. Now, you should see a row for each fact, saved to the _response_body attribute. 

Now that we have isolated the JSON we need, delete the JSONFormatter from the workspace.

10. Rename the API Response Attribute

To keep our workflow organized, it would be better to have a more descriptive attribute name than _response_body. Add and connect an AttributeRenamer transformer to the JSONFragmenter’s Fragments port.  

Inside AttributeRename, select the _response_body as the Input Attribute. Enter Cat Fact as the Output Attribute. Set If Input Attribute Is Missing to Preserve Existing Output Attribute.

If you run the workspace again, you’ll find the facts are now listed under a Cat Fact attribute. Much better.

11. Sample the Data

We want just one random fact to send to a recipient, not all of them. Add a Sampler and connect it to the AttributeRenamer. Open it up and change the Sample Rate (N) to 1. The Sampling Type will be First N Features. Lastly, set the Randomize Sampling parameter to Yes. Now we will get a random cat fact from the API response every time the workspace runs. 

12. Configure the Emailer

Finally, we can use our workflow to send our random cat fact to the recipient of our choice. Add an Emailer and connect it to the Sampler’s Sampled Port. This transformer requires an Gmail Web Connection or SMTP configuration to work.  
 

Add your Email to the From: parameter and a recipient in the To: parameter. I’ll send one first to myself for testing. Add a Subject line, like “Cat Facts!”, and select the Cat Fact attribute as the body. Use the Text Editor to customize your email body. 
 

Choose your Service and set a configuration. In this case, I’ll use my Gmail address and stored Gmail Web Connection. 

Select OK. Your final workspace should look like this: 

13. Check your Email

Run the workspace. Check the inbox of the recipient email address. Inside, you should find a new email named “Cat Facts!” sent from your workspace. 

Every time the workspace runs, the HTTPCaller will make a GET request to the Cat Facts API and receive the most recent data. To get the most out of this workspace, think about setting it up on an FME Flow Schedule to make a daily API call and deliver a random fact to your (or your friend's) inbox.

More on HTTPCaller

This step-by-step guide showed you how to make a simple API call using HTTPCaller and interpret its response. The HTTPCaller has built-in parameters to handle even your most complex API calls from inside FME Workbench. Try another tutorial to get more experience using this transformer in your workflows or learn more about APIs by reading the Getting Started with APIs tutorial. 

Data Attribution 

The data used here originates from data made available by Cat Facts API. Cat Facts is ad-free and available to use without cost.