Offset-Based Pagination Using the FME Flow REST API V4 and OpenAPICaller

Introduction

In FME, there are many ways to connect to and utilize APIs. We can make API calls using a FeatureReader, a JSONExtractor, or an HTTPCaller. In this tutorial, we will cover how to make an FME Flow API call with the OpenAPICaller. This tutorial requires FME 2024.1 and the use of the FME Flow REST API V4. Earlier versions do not have the required transformer. If you are using an earlier version of FME, please see the V3 version of the article: Offset-Based Pagination Using the FME Flow REST API V3, which uses the HTTPCaller

The OpenAPICaller transformer provides users with an easy way to interact with API endpoints by importing an OpenAPI specification, which can be provided as a JSON or YAML file. The transformer will populate a list of available endpoints and automatically populate all the appropriate parameters for the user when an endpoint is selected. This takes the guesswork out of configuring an API endpoint in FME Workbench. 

Requirements

• FME Form and Flow 2024.1+

Step-by-Step Instructions

1. Open a new Workbench

Add a creator to the canvas and then an OpenAPICaller. Connect the creator output to the input of the newly added transformer. 

2. Import OpenAPI Specification 

Double-click on the OpenAPICaller transformer to open the parameters window. The first thing to do is select the OpenAPI to read in. This can be a URL, a JSON file, or a YAML file. Once a specification is loaded, the import status and Select Endpoint panes will be filled.

3. Select Endpoint

The endpoint we want to use is GET /jobs. Scroll down in the Select Endpoint pane until you find /jobs. Click on the heading, and select GET. 

Once the endpoint is chosen, the Query Parameters window will populate with the name, type, and description of each of the parameters. Take note that the only required parameter is the status - we will populate this in the next window. Click Import Request Parameters to close the window and return to the OpenAPICaller Parameters Window.

4. Authenticate the FME Flow Connection

Enable authentication and select the following: 

• Authentication Method: Web Connection
• Web Connection: <your FME Flow name>

Recall that the query string parameter “status” needs a value. Under the Query String Parameters heading, click on the Value column. Type in the status you want to call—for this tutorial, I am using the “success” status. 

• Query String Parameters:
  • status: success

For organization, you can remove all the optional query string parameters. Please keep status, limit, and offset.

At this point, you can click Send Test Request to ensure the connection and endpoint are working. If you get any errors, you can fix those now. Common ones include missing query string parameters or FME Flow connection issues. 

5. Configure the Output

Continuing down in the OpenAPICaller parameters window, you will see the Output Configuration headings. Set the following: 

• Response Handling Mode: Output JSON Fragments as Features
• JSON Query to Fragment: json[“totalCount”]

At this stage, the OpenAPICaller does not need to be configured further. Click OK to finish adding the transformer to the canvas.

6. Run the Workspace

In the Data Preview pane, we can see that we successfully made the call, and the json_index we queried gave us the number of successful jobs completed on our FME Flow in the response body. 

7. Test the totalCount

The query string parameter limit was set to 100 in our call. If there are fewer than 100 results, we only need one call and can parse the JSON response right away. Otherwise, we will need to loop through and make multiple calls.

Add a Tester transformer to the canvas and connect it to the output port of the OpenAPICaller. We want to test the value of totalCount to see if we need to paginate through the results. Set the following parameters:

• Left Value: _response_body
• Operator: >
• Right Value: 100

A passed port on the tester says there are more than 100 results, and the failed port indicates there are fewer than 100 results. 

8. Pagination through Cloning

Add an AttributeManager to the canvas and connect it to the “Passed” port of the Tester. Manage the attributes by removing any that are not required. We will keep the totalCount attribute and add an additional one named cloner.

Set the following parameters in the AttributeManager

• Remove Attribute:
  • Input Attribute: _creation_instance
    • Action: Remove
• Remove Attribute:
  • Input Attribute: _http_status_code
    • Action: Remove
• Remove Attribute:
  • Input Attribute: json_index
    • Action: Remove
• Remove Attribute:
  • Input Attribute: json_type
    • Action: Remove
• Remove Attribute:
  • Input Attribute: message
    • Action: Remove
• Rename Attribute:
  • Input Attribute: response_body
  • Output Attribute: totalCount
    • Action: Rename
• Add Attribute:
  • Output Attribute: cloner
    • Value: @add(@int(@div(@Value(totalCount),100)),1)

The order of the attributes matters here. The totalCount attribute needs to be above the newly created cloner attribute. The value of the cloner attribute references totalCount and will not work if it is placed above totalCount in the list.

The value of the cloner attribute ensures that the value has no decimals (int) after being divided, and includes the original plus the pagination (add, 1).

Click OK to accept the parameters.

9. Add a Cloner Transformer

Connect a Cloner to the output port of the AttributeManager. In the Cloner properties, set:

• General:
  • Number of Copies: cloner

The number of copies created will equal the value in the attribute totalCount. We will use the _copynum attribute.

10. Expose _copynum Attribute and Calculate the Value

Add a second AttributeManager to the canvas and connect it to the Copy port on the Cloner. Multiplying the _copynum by the offset will allow the next OpenAPICaller to make that many calls to FME Flow. In the AttributeManager_2 parameters, set the following: 

• Set Value:
  • Output Attribute: _copynum
    • Value: @mult(@Value(_copynum), 100)

11. Final Call

Copy and paste the OpenAPICaller created at the start of this tutorial. Connect it to the Output port of the AttributeManager_2 transformer. All the parameters we set are still configured, but we need to change the offset query string parameter. We will change this from 100 to _copynum. This will make the OpenAPICaller make as many calls as are required to go through all the pages of jobs on FME Flow.

Double-click on the OpenAPICaller_2 transformer and scroll down in the Query String Parameters box until you find the offset parameter. Update: 

• Offset: _copynum
• JSON Query to Fragment: json[“items”][*]

Click OK to accept the parameters.

12. Add a JSONFlattener Transformer

Add a JSONFlattener to the canvas and connect the input port to the output port of the second HTTPCaller, and the failed output port of the Tester. If there are fewer than 100 jobs returned, the failed port will go straight to the JSONFlattener since pagination isn’t required.

This will parse your data into a list attribute. For more information on lists, see the Getting Started with List Attributes article 

Set the following parameters:

• Source:
  • Input Source: JSON Document
  • JSON Document: _response_body

13. Run the Entire Workspace

Run your workspace, and it should look like the image below.