Getting Started with API Pagination

Evie Lapalme
Evie Lapalme
  • Updated

FME Version

  • FME 2022.0

Introduction

In FME, there are many ways to connect and utilize APIs. We can make API calls using a FeatureReader, a JSONExtractor, or an HTTPCaller. In this tutorial series we will be covering how to make several calls where pagination is required.

What is a REST API?

API (Application Programming Interface) is a software intermediary that allows two applications to talk to each other over the web. The RESTful API allows the use of HTTP requests to access and use data such as read, update, or even create by calling a GET or a POST. You can create your own custom REST API workflows using the HTTPCaller with authorization with a token or web connection!

 

What is Pagination?

Pagination is the process of separating print or digital content into discrete pages. With an API call most responses are in JSON where pagination refers to displaying a portion of data for a large dataset. This could mean the first 20 items, thousand items, or whatever the page limit is set to. 

 

When to Loop with FME?

A loop is often used to carry out iteration; where a process repeats to gradually narrow the process to the desired result. A loop is linked to a condition; i.e. the action continues until a certain condition is met. This is the most common use of a loop in FME because iterations are built into processing features automatically. You can only create loops with FME in custom transformers. For more information on loops see To Loop or Not to Loop webinar.

 

Why Use FME to use Pagination?

  • Integrate your existing data with API data
  • Create and update with API data automatically
  • Accumulate all available data from the API call
  • Automate your workflow

 

Articles

Cursor-Based Pagination

APIs that use cursor-based pagination will return an ID in the response that points to the next item in the dataset, or ‘next cursor’. A user provides the next cursor in the subsequent request to get the next batch of records until no next cursor is returned by the API, meaning there are no further records to fetch. 

This article will demonstrate how to call an API that uses cursor-based pagination in FME Workbench.

Offset-Based Pagination

APIs that use offset and limit based pagination will sometimes return a page number, a total record number, or nothing at all. To cycle through the pages we use the offset to determine the end of the limit to reach the next page. The end of the fetching ends when there is nothing returned or when the total records arrive. 

In this tutorial we will be covering how to make an FME Server API call and Yelp Fusion API call with the HTTPCaller where pagination is required because the amount of records exceeds the page limit.

 

Additional Resources

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.