Introduction
In FME, there are many ways to connect to and utilize APIs. We can make API calls using a FeatureReader, a JSONExtractor, or an HTTPCaller. In this tutorial series, we will cover how to make several calls where pagination is required.
What is a REST API?
API (Application Programming Interface) is a software intermediary that allows two applications to talk to each other over the web. The RESTful API allows the use of HTTP requests to access and use data, such as read, update, or even create, by calling a GET or a POST. You can create your own custom REST API workflows using the HTTPCaller with authorization with a token or web connection!
What is Pagination?
Pagination is the process of separating print or digital content into discrete pages. With an API call most responses are in JSON where pagination refers to displaying a portion of data for a large dataset. This could mean the first 20 items, thousand items, or whatever the page limit is set to.
When to Loop with FME?
A loop is often used to carry out iteration, where a process repeats to gradually narrow the process to the desired result. A loop is linked to a condition; i.e., the action continues until a certain condition is met. This is the most common use of a loop in FME because iterations are built into processing features automatically. You can only create loops with FME in custom transformers. For more information on loops, see the To Loop or Not to Loop webinar.
Why use FME for Pagination?
- Integrate your existing data with API data
- Create and update with API data automatically
- Accumulate all available data from the API call
- Automate your workflow
Articles
Cursor-Based Pagination
APIs that use cursor-based pagination will return an ID in the response that points to the next item in the dataset, or ‘next cursor’. A user provides the next cursor in the subsequent request to get the next batch of records until no next cursor is returned by the API, meaning there are no further records to fetch.
This article will demonstrate how to call an API that uses cursor-based pagination in FME Workbench.
Offset-Based Pagination
APIs that use offset and limit-based pagination will sometimes return a page number, a total record number, or nothing at all. To cycle through the pages, we use the offset to determine the end of the limit to reach the next page. The fetching ends when nothing is returned or when the total records arrive.
Offset-Based Pagination Using the FME Flow REST API V3 - This tutorial uses the HTTPCaller and the V3 REST API. Please use this tutorial if you are using FME 2024.0 and earlier
Offset-Based Pagination Using the FME Flow REST API V4 and OpenAPICaller—This tutorial uses the OpenAPICaller transformer and requires FME 2024.1 and newer