Introduction
As web applications and devices continue to become more connected, more real-time data is produced. The FME platform offers several solutions for ingesting real-time data, including webhook triggers, WebSocket receivers, queue connectors, and streams, to meet the requirements for your real-time data project depending on the systems, frequency, and size of data you’re working with.This page covers some of the available approaches, common formats, tutorials, and resources to get started with real-time data and FME.
Content Overview
- Terminology
- Automations vs Streams
- Tutorials
- Format Documentation
- Additional Resources
- Troubleshooting
- Known Issues
- Support
Terminology
API (Application Programming Interface): A set of defined constraints that allows the software to communicate with a defined protocol, like HTTP.Authentication: The process of verifying who a user is.
Authorization: The process of checking what information and applications a user has permission to access.
FME Engine: Processes job requests by running FME workspaces. Each FME Engine processes a single job request at a time.
JSON/XML/GML: Common file formats for transmitting data across the web.
Kafka: An open-source message broker created by Apache.
Message Broker: Intermediary software that translates messages between sending and receiving protocols.
Message Queue: A component (e.g. a network node) that allows for asynchronous service-to-service communication. Messages are stored in the queue until they are processed and deleted. Each message is processed only once, by a single consumer.
Server: Receives the request from the client (e.g. a web site) and returns data.
Token: An alpha-numeric string used to authenticate and identify an application to an API as opposed to defining a user.
Web Connection: A convenient and secure way to store and reuse connection parameters to various web services.
Webhook: An event-driven message sent over HTTP.
Web Service: An application that listens for requests, communicates with other client or server applications, and transmits data across the web via standard protocols like HTTP or HTTPS.
WebSocket: A protocol implemented on top of HTTP, that allows for bidirectional, browser-based client-server communication.
Automations vs Streams
FME Flow (formerly FME Server) provides two powerful solutions for ingesting real-time data: Automations and Streams.FME Flow Automations can be triggered by event data received from another system via a WebSocket or webhook message, typically in JSON format. This method is effective if events in the source system are well-defined, for example, a form submission, and data throughput is not excessive. A message up to every few seconds is generally acceptable, depending on the size of the data being received. With automations, an engine on FME Flow will only be in use when a message is being processed. Note that webhooks and the FME Flow REST API can also trigger an FME Workspace directly. Please see Automating Workflow Submission from a Third Party Application To FME Flow for an overview of these options.
Suppose the system you are consuming data from can't connect via a WebSocket or webhook. In that case, you can still use other FME Flow Automation triggers to perform near real-time processing by frequently polling. For example, you can poll an email server for new emails or a network drive for new files and trigger an automation when one is received.
If your real-time data is arriving in a constant flow of many messages per second, up to thousands or even hundreds of thousands, then FME Flow Streams are the solution. For example, high-volume data readings from devices like sensors. A stream will allow you to keep a workspace running continuously to constantly receive a data stream from a message broker, like Kafka, MQTT or RabbitMQ. Streams can also use WebSocket connections.
The image below illustrates the difference between automations and streams. Watch this video for a summary of event data vs stream data.
Which Approach is Right for Me?
The table below helps you compare and contrast automations versus streams to understand which approach is right for your use case.Automation | Stream | |
---|---|---|
Data throughput | Designed for low-volume data throughput. Up to one message per second may be ok depending on the size of the messages being processed. | Designed for high-volume data throughput. Can process thousands of messages per second. |
Data source | Other web applications via webhooks or WebSockets. | Message brokers or WebSocket connections. |
FME Engine impact | Engines are only in use when processing Run Workspace actions within an automation. | The engines assigned to the stream are always in use, as they are intended to be constantly processing data. Your server must have at least one engine dedicated to processing the stream. CPU-Usage Engines are recommended for streams that may be idle for long periods of time. |
Job orchestration | Allows you to build complex workflows made up of several FME Workspaces and other internal and external actions running in sequence or in parallel. | One FME Workspace running constantly per stream. Intended for simple processing, due to the high volume of data. |
Tutorials
Automations
Taking Action on Real-Time Data with FME (Websockets)- Real-Time Situational Awareness (Flood Monitoring) - Part 1
- Real-Time Situational Awareness (Flood Monitoring) - Part 2
- Push Data from ArcGIS Survey123 to an Application in Real-time with Webhooks
- Push Data from ArcGIS Field Maps to an Application in Real-time with Webhooks
- Tutorial: Getting Started with Cityworks
Streams
FME and Stream Processing landing pageDatabase Webhooks
The articles below cover automating real-time database updates to FME Flow from different databases.- PostgreSQL Single Edits
- PostgreSQL Bulk Edits
- SQL Server Single Edits
- SQL Server Bulk Edits
- Oracle Single Edits
- Oracle Bulk Edits
Other Real-Time Tutorials
The articles below are no longer updated and were written before FME Flow Automations and Streams, but may still be of interest.PostGIS to FME Server in Real Time (Node.js) Tutorial
- HTTP POST Requests to FME Server from Node.js
- Streaming features from PostGIS to FME using WebSockets
Format Documentation
FME has built-in tools to create real-time solutions.Automations
Popular transformers for working with webhooks and WebSockets in FME Workbench include: Automation triggers for receiving real-time event data include:Streams
Popular transformers for working with streaming data in FME Workbench include:- AWS IoT Connector
- Azure IoT Connector
- Azure Event Hubs Connector
- Azure Queue Storage Connector
- Google Big Query Connector
- Google Cloud Pub Sub Connector
- Kafka Connector
- MQTTConnector
- Rabbit MQ Connector
- SQS Connector
- Websocket Receiver
Additional Resources
An understanding of web services is helpful when exploring automations and streams for real-time data. Please see Working with Web Services and FME .
Automations
Blogs- Connect to APIs and Webhooks in No Time
- Reeling in ArcGIS Online Webhooks with FME
- Bringing Real-Time Data to the Web – A Live Example using Node.js, Pusher, and FME Server
- Don’t Hate, Automate! The Fundamentals of Automations [0:59:24]
- Hook, Line, and Sinker Reeling in ArcGIS Webhooks [0:55:43]
Streams
Blogs Webinars- Empowering Real-Time Decision Making with Data Streaming [1:13:45]
- Introduction to Data Stream Processing [0:50:43]
- Full Stream Ahead: Authoring Workflows for Scalable Stream Processing [1:03:53]
- What is Stream Data Processing [0:07:07]
- Publish a Workspace for Streaming in FME [0:03:28]
- Stream Processing: Window Data Streams [0:04:11]
- Stream Processing: Join Data Depending on Attribute Values [0:02:42]
- Stream Processing: Data Enrichment Based Upon Location [0:02:50]
- Stream Processing: Select Based on Location [0:03:12]
- Stream Processing: Geofencing [0:03:55]
- Stream Processing: Event Detection Based on Location [0:06:40]
- Stream Processing: Calculating Movement Statistics [0:05:55]
- Stream Processing: Snap Data to a Network [0:06:18]
- Stream Processing: Summarize Data Through Aggregation [0:07:39]
- Stream Processing: Remove Duplicate Data Within a Certain Time [0:05:38]
Comments
0 comments
Please sign in to leave a comment.