Getting Started with FME Form: Working with Web Data

Introduction 

In this article, you'll discover how to create a workspace in FME Workbench, the authoring application that accompanies FME Form, with a specific emphasis on utilizing web data. Follow along with the step-by-step instructions to learn how to trigger a workspace with a Creator transformer, get data from an API using the HTTPCaller, and then view the data in a human-readable format in Visual Preview. 

FME Desktop was rebranded to FME Form in 2023. FME 2024.0 features a new interface. This tutorial can be used with any version of FME. The concepts are the same, but the screenshot appearances may differ.

Source Data

The source dataset is a road closure dataset that can be retrieved via an API. 

 

Step-by-step Instructions

1. Start a Blank Workspace
Start FME Workbench, then click the Blank Workspace button.


The FME Workbench canvas is where the workspace will be built using readers, writers, and transformers. 


2. Start (Trigger) the Workflow 
Normally, the first step when building a workspace is adding a reader component, which triggers the flow of data, but interacting with the web is different. The workspace still needs an initial trigger to initiate the flow of data, so instead of a reader, we will add a Creator transformer to the canvas. 

Transformers are the building blocks within a workspace to manipulate data, and each has a specific function. Click on the canvas and then start typing Creator to bring up the Quick Add dialog. The Quick Add dialog is where readers, writers, and transformers can be added. Double-click on the Creator under FME Transformers to add it to the workspace. 



3. Add an HTTPCaller and Connect Workflow
To make a call to a URL within FME, an HTTPCaller transformer is used. Add an HTTPCaller to the canvas the same way the Creator was added. 


When the workspace is run (initiated) in a later step, the Creator will send out a null feature to trigger the workspace. For the Creator to trigger the HTTPCaller, they need to be connected. Click and drag from the arrow on the Creator to the red arrow on the HTTPCaller. This connection line sends data from the Creator into the transformer for processing.


4. Modify Transformer Parameters
The HTTPCaller transformer can access content from a URL, such as an API. In this example, we will get road closure information from an API. Double-click anywhere on the HTTPCaller transformer to open the parameters. 


Copy the following URL, and then paste it into the red box next to Request URL. 

https://opendata.vancouver.ca/api/explore/v2.1/catalog/datasets/road-ahead-current-road-closures/records?limit=5

The API will pull five of today's road closures for Vancouver, British Columbia, Canada. 

Note: In FME 2023.2 and prior, the example API was for weather, but that API has since gone offline. Please use the above road closures API. The steps are the same. 

Next, since we want to retrieve (GET) the weather from the API, we need to set the HTTP Method. Click on the drop-down arrow inside of the red box next to HTTP Method to open the menu, then select GET. 


There are other parameters that can be set, such as query string, headers, and authorization. However, for this example, we will leave them as the default. Click OK to save the HTTPCaller parameters and close the dialog. 

5. Run the Workspace
We can now run the workspace, which will GET the response from the road closure API. Click on the green Run button on the top toolbar (ribbon). Clicking Run will trigger the Creator to create a single null feature, and the HTTPCaller will send out a single GET call, then the response received back will be stored in a new attribute created by the HTTPCaller called _response_body. 


6. View Response  
The response from the API can be viewed in Visual Preview, which is the data viewer within FME Workbench. Click on the green eye icon on the HTTPCaller Output port to view the response. 

Note: In FME 2023.2 and prior, the cache icon is a magnifying glass. 

In the Visual Preview window, you will see a table of attributes. When working with web data, the _response_body contains the response back from the URL that was accessed, and the _http_status_code will return the status of the call. Double-click on the cell below _response_body to open the response in a larger window. 


In the larger window, we can change the syntax highlighting to make the response more human-readable. In the bottom left corner, click on the ABCXYZ button and select JSON from the drop-down. 


Now we can easily see five road closures in Vancouver, British Columbia. 


7. Continue Workflow
Since web data varies greatly in both structure and format, the next steps after the HTTPCaller will be different. 

This API returned data as JSON, so the next step is to flatten the JSON. See Transforming JSON using the JSONExtractor, JSONFlattener, and JSONFragmenter.

 

Continue Your FME Journey  

• Getting Started with FME Form: Working with Table/Tabular Data
• Getting Started with FME Form: Working with Databases
• Getting Started with FME Form: Working with Spatial Data

Want to learn about a specific format?

• Tutorial: Getting Started with JSON
• Tutorial: Getting Started with APIs
• Tutorial: Getting Started with XML

Want to continue learning FME Form basics?

• The FME Academy is a guided, on-demand way to learn the entire FME Platform, and it's FREE! 

Interested in self-serve and automation?

• Tutorial: Getting Started with FME Flow
• FME Flow Authoring - FME Academy

Have a specific question?

• Reach out to the FME Community 

Data Attribution

The data used here originates from data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.