Working with Real-Time Data and FME

Matt Meeboer
Matt Meeboer
  • Updated

Introduction

As web applications and devices continue to become more connected, more real-time data is produced. The FME platform offers several solutions for ingesting real-time data, including webhook triggers, WebSocket receivers, queue connectors, and streams, to meet the requirements for your real-time data project depending on the systems, frequency, and size of data you’re working with. 

This page covers some of the available approaches, common formats, tutorials, and resources to get started with real-time data and FME.  
 

Content Overview 

 

Terminology

API (Application Programming Interface): A set of defined constraints that allows the software to communicate with a defined protocol, like HTTP.  

Authentication: The process of verifying who a user is.

Authorization: The process of checking what information and applications a user has permission to access.

FME Engine: Processes job requests by running FME workspaces. Each FME Engine processes a single job request at a time.

JSON/XML/GML: Common file formats for transmitting data across the web.

Kafka: An open-source message broker created by Apache.

Message Broker: Intermediary software that translates messages between sending and receiving protocols.

Message Queue: A component (e.g. a network node) that allows for asynchronous service-to-service communication. Messages are stored in the queue until they are processed and deleted. Each message is processed only once, by a single consumer.

Server: Receives the request from the client (e.g. a web site) and returns data. 

Token: An alpha-numeric string used to authenticate and identify an application to an API as opposed to defining a user.

Web Connection: A convenient and secure way to store and reuse connection parameters to various web services.

Webhook: An event-driven message sent over HTTP.

Web Service: An application that listens for requests, communicates with other client or server applications, and transmits data across the web via standard protocols like HTTP or HTTPS.

WebSocket: A protocol implemented on top of HTTP, that allows for bidirectional, browser-based client-server communication.
 

Automations vs Streams

FME Flow (formerly FME Server) provides two powerful solutions for ingesting real-time data: Automations and Streams.

FME Flow Automations can be triggered by event data received from another system via a WebSocket or webhook message, typically in JSON format. This method is effective if events in the source system are well-defined, for example, a form submission, and data throughput is not excessive. A message up to every few seconds is generally acceptable, depending on the size of the data being received. With automations, an engine on FME Flow will only be in use when a message is being processed. Note that webhooks and the FME Flow REST API can also trigger an FME Workspace directly. Please see Automating Workflow Submission from a Third Party Application To FME Flow for an overview of these options.

Suppose the system you are consuming data from can't connect via a WebSocket or webhook. In that case, you can still use other FME Flow Automation triggers to perform near real-time processing by frequently polling. For example, you can poll an email server for new emails or a network drive for new files and trigger an automation when one is received.

If your real-time data is arriving in a constant flow of many messages per second, up to thousands or even hundreds of thousands, then FME Flow Streams are the solution. For example, high-volume data readings from devices like sensors. A stream will allow you to keep a workspace running continuously to constantly receive a data stream from a message broker, like Kafka, MQTT or RabbitMQ. Streams can also use WebSocket connections.

The image below illustrates the difference between automations and streams. Watch this video for a summary of event data vs stream data.

Automations vs Streams
Which Approach is Right for Me?

The table below helps you compare and contrast automations versus streams to understand which approach is right for your use case.
 
 AutomationStream
Data throughputDesigned for low-volume data throughput. Up to one message per second may be ok depending on the size of the messages being processed.Designed for high-volume data throughput. Can process thousands of messages per second.
Data sourceOther web applications via webhooks or WebSockets.Message brokers or WebSocket connections.
FME Engine impactEngines are only in use when processing Run Workspace actions within an automation.The engines assigned to the stream are always in use, as they are intended to be constantly processing data. Your server must have at least one engine dedicated to processing the stream. CPU-Usage Engines are recommended for streams that may be idle for long periods of time.
Job orchestrationAllows you to build complex workflows made up of several FME Workspaces and other internal and external actions running in sequence or in parallel.One FME Workspace running constantly per stream. Intended for simple processing, due to the high volume of data.


Tutorials

Automations

Taking Action on Real-Time Data with FME (Websockets) Working with FME and Webhooks  

Streams

FME and Stream Processing landing page
 

Database Webhooks

The articles below cover automating real-time database updates to FME Flow from different databases.  

Other Real-Time Tutorials

The articles below are no longer updated and were written before FME Flow Automations and Streams, but may still be of interest.

PostGIS to FME Server in Real Time (Node.js) Tutorial  

Format Documentation

FME has built-in tools to create real-time solutions. 
 

Automations

Popular transformers for working with webhooks and WebSockets in FME Workbench include: Automation triggers for receiving real-time event data include:  

Streams

Popular transformers for working with streaming data in FME Workbench include:  

Additional Resources


An understanding of web services is helpful when exploring automations and streams for real-time data. Please see Working with Web Services and FME .
  

Automations

Blogs Webinars Demo Videos  

Streams

Blogs Webinars Demo Videos  

Troubleshooting

FME Flow Troubleshooting Guide  

Known Issues

For a list of Known Issues, see the following articles: 2023.x2022.x, 2021.x 
 

Support

Can’t find what you are looking for? Search our Community for a related question or post your question in our Forums or contact Support.
 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.