Introduction
The HTTPCaller in FME Workbench lets you make REST API calls to get data from external applications. Often, API developers will limit the number of records returned in a response to make it easier to handle. A user must then make subsequent requests to get all records. This is called pagination. There are several types of pagination, but this article will focus on cursor-based pagination.
APIs that use cursor-based pagination will return an ID in the response that points to the next item in the dataset, or ‘next cursor’. A user provides the next cursor in the subsequent request to get the next batch of records, continuing until no next cursor is returned by the API, indicating there are no further records to fetch. A user can also provide the number of records, or page size, they want to retrieve with each fetch. The page size is limited by the API developer to prevent responses from becoming too large and defeating the purpose of pagination.
This article uses the new native looping functionality introduced in FME 2026.2. If you are using a version of FME prior to 2026.2, please see Cursor Based API Pagination 2025.2 and older
Cursor-based pagination is more scalable than offset pagination, making it ideal for larger datasets. It simply uses the next cursor ID to query and get the next set of records. Offset pagination returns the total number of records, the total number of pages, and the current page with each request. While this information can be valuable, it requires far more processing.
In the following tutorial, we’ll look at an example of calling an API that uses cursor-based pagination, Slack’s conversation history API, using FME Workbench to recursively get all messages. This logic we’ll walk through has been published in the SlackMessageReader custom transformer to the FME Hub. You can use the transformer as a guide to create your own API calls that use pagination. It also contains more parameters and functionality than we will delve into in this guide.
Requirements
- This tutorial requires a minimum FME version of 2026.2. If you are using an earlier version, please see: Cursor-Based API Pagination
- A Slack bot app that has been added to the Slack workplace and channel from which you will fetch messages
- The Slack bot has been granted the necessary permissions and scopes to use the API
- The bot token from your Slack app
Step-by-Step Instructions
Part 1: Create the API Call
1. Open a new workspace
In FME Workbench, create a new workspace. Add a Creator transformer.
2. Add User Parameters
Let’s add some user parameters that will hold our token and Slack channel ID.
From the Navigator, Right-click User Parameters and then select Manage User Parameters
Click the ‘+’ button and add a text parameter.
Enter the following:
- Parameter Identifier: token
- Label: Token
Add a second text parameter and enter the following:
- Parameter Identifier: channelID
- Label: Slack Channel ID
Click OK to save and close the User Parameters.
3. Configure Slack API Call
Add an HTTPCaller to the canvas and connect it to the creator. Configure the HTTPCaller as follows:
- Request URL: https://slack.com/api/conversations.history
- HTTP Method: GET
-
Query String Parameters:
- channel: channelID parameter
- limit: 200 (the maximum page size recommended by Slack)
- cursor: do not enter a value, we will set this later
-
Headers:
- Authorization: Type the word Bearer, then a space, and then the token parameter $(token)
- Response Body Attribute: _response_body
-
Maximum Number of Concurrent HTTP Requests: 1
- This value must be 1 when looping
Click OK
There are additional parameters that could be added to give more control over the results returned from the Slack API, but for the purpose of this example, we will just use these. For a more detailed example, see the SlackMessageReader in the FME Hub.
4. Test the API Call
Run the workspace, and it will prompt you to enter your Token and Slack ID Channel.
- Token: the bot token (starting with xoxb-) from your Slack app configuration
- Slack Channel ID: the channel ID of the Slack channel that your bot has been added to
Run the workspace and inspect the HTTPCaller output in Data Preview. There is one feature with a _response_body attribute containing the first 200 Slack messages in JSON format (since we set the API limit to 200).
Let’s inspect the structure of the JSON response a little further. Click the ellipses [...] next to the _response_body value to view the full response.
Format as JSON.
We see that most of the JSON response consists of an array of the first 200 messages in the Slack channel.
Scroll to the bottom of the response. There is another object called ‘response_metadata’ that contains the next_cursor ID. This is the value we need to pass to the API to get the next 200 messages.
Now, we need to create a loop to get all messages from Slack using cursor-based pagination.
Part 2: Create a Cursor-Based Pagination Loop
1. Separate the JSON Keys in the Response
Add a JSONFragmenter to the canvas and connect it to the HTTPCaller output. Configure the parameters:
- JSON Attribute: _response_body
- Flatten Query Results into Attributes: Yes
- Recursively Flatten Objects/Arrays: No
Other fields can remain with their defaults.
Click OK. Run the workspace. Inspect the results in Visual Preview. There are seven JSON keys that have been output, but we only care about two: messages and response_metadata.
2. Filter the JSON Keys
Add a TestFilter to the canvas and connect it to the JSONFragmenter’s output. Add two conditions to route the JSON keys to different outputs:
-
Test If:
@Value(json_index) = messages- Output Port: messages
-
Test Else If:
@Value(json_index) = response_metadata- Output Port: response_metadata
Click OK. Your workspace should look like the image below. Run the workspace.
Inspect the TestFilter’s output. There is now one feature for all messages and another for response_metadata that contains the next_cursor. It is this second feature we will use to drive the pagination loop. The remaining features are not needed and are sent to the unfiltered port.
Place a bookmark around the HTTPCaller, JSONFragmenter and the TestFilter and call it ‘Slack API Call’.
3. Parse the Response Metadata JSON
Connect a JSONFlattener to the response_metadata output port. Set the JSON Document to _response_body and, in the Attributes to Expose, type next_cursor. Other fields can remain with their defaults. Click OK.
Run the workspace again and inspect the JSONFragmenter output. We now have a next_cursor attribute on the response_metadata feature to pass to the API.
4. Create the Pagination Loop
Select all transformers except for the Creator, and then click Loop on the toolbar.
Once the book block is added to the canvas, connect the output of the JSONFlattener to the loop port on the bottom right of the loop block and connect the loop port on the bottom left to the input of the HTTPCaller. Your workspace should look like the screenshot below.
Now that the loop has been set up and the feature with the cached next_cursor attribute is connected to the HTTPCaller, open the HTTPCaller parameters again and set the cursor parameter to the next_cursor attribute that was exposed.
Lastly, put a Logger transformer outside of the loop and connect it to the JSONFlattener output.
5. Test the Loop
We’re now ready to test the pagination loop!
The Slack API will return the response_metadata JSON key, which contains the next_cursor, as long as there are additional messages to fetch. Once there are no more, no response_metadata key will be returned from Slack, and thus no feature will be output from the TestFilter’s response_metadata port, ending the loop. Each time the API is called, we will get 200 messages (except on the last call) and output them all as a single feature, for now.
Run the workspace. Depending on how many messages are in your Slack channel, it may take a couple of minutes to run. Once the workspace has finished running, inspect the Logger output. Each feature contains a JSON array of 200 messages (except the last call).
Take a look at the _response_body for some of the features; they should all contain a different set of messages. The count of features output is the total number of times that the Slack API was called. The iteration count at the top of our loop block is one more than our logger because the loop ran one additional time, and did not find any more slack messages during the last iteration. Our cursor-based pagination loop is working!
Part 3: Parse the JSON into Features
1. Parse the Message JSON
Increase the size of the loop block and add a JSONFragmenter and connect it to the TestFilter's messages output port.
- Input Source: JSON Attribute
- JSON Attribute: _response_body
- Flatten Query Results into Attributes: Yes
- Recursively Flatten Objects/Arrays: No
We aren’t recursively flattening in this case because Slack sends a lot of data for each message, like attachments and reactions. We’ll leave those as JSON objects for now, and just extract the key message attributes.
Run the workspace. Each message from each API call is now output as a feature. Our 6 features have become over 1,300.
2. Clean up Attributes and Workspace
Add an AttributeManager to the canvas and connect it to the JSONFragmenter output. Remove all currently exposed attributes and expose the message text, user, and team instead.
Finally, let’s clean up the workspace a bit. Add a bookmark around the JSONFlattener and name it ‘Get the Next Cursor’. Add another bookmark around the second JSONFragmenter and AttributeManager and name it ‘Fragment Slack Messages’. The workspace should now look like this.
Run the workspace. Inspect the Logger output in Visual Preview. The Slack message text, user, and team are all displayed. Additional attributes can be exposed as needed.
This approach can be used for other API calls that require cursor-based pagination.
Additional Resources