Streaming IoT Data from a REST API in FME with the EndlessLooper

Introduction

FME supports message brokers typically used to connect to IoT data sources, such as Kafka, MQTT, and RabbitMQ. However, real-time data can also originate from REST APIs. This could include providers such as GeoTab, Samsara, and more.

To create a workflow that continuously polls an API endpoint, users can use the EndlessLooper transformer available on the FME Hub. The EndlessLooper transformer, when paired with the HTTPCaller transformer, can create a workflow that runs indefinitely, similar to a workflow that uses a message broker transformer. 

The EndlessLooper transformer contains a parameter called Delay Per Feature (Seconds), which dictates how often a feature exits the EndlessLooper. 

This parameter can be used to control how often the API is being polled. 

Why do we need the EndlessLooper?

The intention is to publish this workspace to FME Flow and then run it indefinitely in an FME Flow Stream. The EndlessLooper allows the workspace to run indefinitely and continuously poll data from the API endpoint instead of restarting the workspace every time it finishes running.

Example

Polling Vehicle Data a REST API

The Massachusetts Bay Transportation Authority (MBTA) offers a public REST API endpoint for retrieving fast and easy access to transportation data—documentation is here. We will use this as our sample data for the following example.

In a scenario where we want to continuously poll the vehicles endpoint for vehicle location information and status, we can create a workspace that uses the EndlessLooper and HTTPCaller transformers. The Delay Per Feature on the EndlessLooper is set to 1 second, and the rate limiting is set on the HTTPCaller to ensure that the rate limit for the API is not reached. 

Additional transformers such as the JSONFragmenter, VertexCreator, and CoordinateSystemSetter are used to extract attribute values and create geometry to be visualized.

Features are then written to the desired destination, in this example, a SpatiaLite database table. On the SpatiaLite writer, the Features Per Transaction parameter is set to 50 to ensure that features are continuously updated in the table. The complete workspace, in FME Form, can be seen below. The workspace can also be downloaded from this article.

Now that the workspace is complete, it can be saved and published to FME Flow, where it can be set up in an FME Flow Stream that will run the workspace indefinitely, continuously updating the SpatiaLite database table with the latest vehicle tracking information to display on a web map or be used for further analysis. For instructions on how to work with streams in FME Flow, see How to use the FME Flow Streams Interface.

Data Attribution

The data referenced in this article originates from the MTBA REST API, provided by MassDOT. Documentation for the API can be found here - https://www.mbta.com/developers/v3-api