Offset-Based Pagination Using the FME Flow REST API V3

As of FME Flow 2025.1, the V3 REST API has been deprecated. The current API will remain accessible, but no new features or general bug fixes will be implemented. Starting with FME Flow 2026.1, the V3 API will be removed. We encourage all users to prepare their systems for this change. 

The V4 REST API offers enhanced security and usability, including new and reorganized endpoints, as well as the removal of certain endpoints. For information on migrating from V3 to V4, please see the FME Flow REST API V4 documentation and click on the heading Migrating from REST API V4.
 

Introduction

In FME, there are many ways to connect to and utilize APIs. We can make API calls using a FeatureReader, a JSONExtractor, or an HTTPCaller. In this tutorial, we will cover how to make an FME Flow API call with the HTTPCaller, where pagination is required because the number of records exceeds the limit. 

This article assumes that you have some knowledge about API calls and using the HTTPCaller. For more information on using the HTTPCaller and using HTTP Requests, see the HTTP Requests with the HTTPCaller article. 

What is a REST API?

API (Application Programming Interface) is a software intermediary that allows two applications to talk to each other over the web. The RESTful API allows the use of HTTP requests to access and use data, such as read, update, or even create by calling a GET, a PUT, or a POST. You can create your own custom REST API workflows using the HTTPCaller with authorization with a token or web connection!

What is Pagination?

Pagination is the process of separating print or digital content into discrete pages. With an API call, most responses are in JSON, and pagination refers to displaying a portion of data for a large dataset. This could mean the first 20 items, a thousand items, or whatever the page limit is set to. 

When to Loop with FME?

A loop is often used to carry out iteration, where a process repeats to gradually narrow the process to the desired result. A loop is linked to a condition; i.e., the action continues until a certain condition is met. This is the most common use of a loop in FME because iterations are built into processing features automatically. You can only create loops with FME in custom transformers. For more information on loops, see To Loop or Not to Loop webinar.

Requirements

• FME Form and FME Flow

Step-By-Step Instructions

This example uses the FME Flow REST API. Since it returns the total count, there is no need to loop. The following example shows how to use a Cloner to get all results from the API

1. Open FME Workbench

Open FME Workbench and click New to create a new workspace. Add a Creator to the canvas.

2. Add an HTTPCaller

Add an HTTPCaller to the canvas and connect it to the Creator. Click on the cog wheel to configure the parameters; they should look like this:

• Request:
  • Request URL: http://localhost/fmerest/v3/transformations/jobs/completed
  • HTTP Method: GET
• Use Authentication: checked
  • Authentication Method: Web Connection
  • Web Connection: Your FME Flow connection
• Query String Parameters:
  • limit: 1000
  • offset: 0
• Headers:
  • Accept: application/json

Click OK to finish configuring the HTTPCaller

3. Extract the Total

Add a JSONExtractor to the canvas and connect it to the output port of the HTTPCaller. Open the parameters and set the following:

• Input Source: JSON Document
• JSON Document: _response_body
• Extract Queries:
  • Target Attribute: totalCount
  • JSON Query: json["totalCount"]

4. Test the totalCount

The response has a maximum limit of 1000; therefore, if there are fewer than 1000 responses, we only need one call and no pagination, and can parse the JSON right away. If there are more than 1000 responses, then we need pagination. To determine if we need pagination, we can use a Tester transformer. Add a Tester to the canvas and connect it to the JSONExtractor. 

In the parameters, enter this logic:

• Left Value: totalCount
• Operator: >
• Right Value: 1000

Click OK to accept the parameters

5. Pagination through Cloning

Add an AttributeManager to the canvas and connect it to the “Passed” port of the Tester. Manage the attributes by removing any that are not required. We will keep the totalCount attribute and add an additional one named cloner.

Set the following parameters in the AttributeManager

• Remove Attribute:
  • Input Attribuite:_creation_instance
    • Action: Remove
• Remove Attribute:
  • Input Attribute: response_body
    • Action: Remove
• Remove Attribute:
  • Input Attribuite:_http_status_code
    • Action: Remove
• Add Attribute:
  • Output Attribute: cloner
    • Value: @add(@int(@div(@Value(totalCount),1000)),1)

The value of the cloner attribute ensures that the value has no decimals (int) after being divided, and includes the original plus the pagination (add, 1).

Click OK to accept the parameters.

6. Add a Cloner to the Canvas

The number of copies will equal the attribute cloner. To achieve this, we will use a Cloner. 



Add another AttributeManager to the canvas and connect it to the Cloner output port. We need to set the value of the _copynum attribute to multiply the value by a thousand. This will be the offset for the HTTPCaller.

• Set Value:
  • Output Attribute: _copynum
    • Value: @mult(@Value(_copynum), 1000)

7. Final Call

Duplicate the HTTPCaller, and connect it to the last AttributeManager. Change the query string parameter offset to the newly created _copynum attribute.

8. Add a JSONFlattener

Add a JSONFlattener to the canvas and connect the input port to the output port of the second HTTPCaller, and the failed output port of the Tester. This will parse your data into a list attribute. For more information on lists, see the Getting Started with List Attributes article 

9. Run the Entire Workspace

Run your workspace, and it should look like the image below.