As of FME 2024.0, support for Esri ArcGIS Online (AGOL), ArcGIS Enterprise Portal, and ArcGIS Server Feature Service formats has transitioned to a unified approach using the new Esri ArcGIS Feature Service (Format). This format replaces the legacy reader/writer formats for each of the three services.
Starting in FME 2026.1, the legacy formats will be hidden in the Quick Add menu. Instead, equivalent functionality is now provided by the downloadable Esri ArcGIS Connector package, available on FME Hub. The package also includes new web services and key transformers such as the ArcGISOnlineConnector, ArcGISAttachmentConnector, and ArcGISBranchVersionManager. The package can be installed directly through the Quick Add menu in FME Workbench. Once installed, the new Esri Feature Service format will appear in the Gallery as a unified method for interacting with all three types of Esri ArcGIS Feature Services.
For details on this transition, including guidance on updating existing workspaces, please refer to the article: Working with Esri ArcGIS Feature Services in FME.
Introduction
The FME Platform makes it easy to integrate real-time data into workflows. Real-time data allows us to act quickly on sudden changes. This article provides a high-level overview of how to process and act on real-time data using FME Flow by examining a fully automated flood monitoring system.
This demo uses data from the U.S. Geological Survey to produce a live-updating ArcGIS Online Dashboard and an email notification service, all made possible by FME. With FME, we can seamlessly connect real-time data to these applications. Whether it’s a third-party source or your sensors, FME can manipulate and process data in real-time.
This article was written in an older version of FME. The concepts and steps are still current, but the screenshots might look different.
What is Real-Time Data?
Real-time data is data generated and delivered for immediate processing. This data often comes from sensors, mobile devices, social media, and applications. Examples include river flood gauges, equipment temperature sensors, your phone's location, or your web server's status.
Real-time data can be classified as event data, which is generated when defined conditions arise, or as stream data, which is continuous and has no discrete beginning or end. This demo focuses on stream data, using a real-time flood monitoring system as an example.
Creating a Real-Time Flood Monitoring System with FME
This will provide an overview of how FME was used to create a real-time flood monitoring system. The goal was to create an email notification system that automatically reacts to incoming real-time water level data and an ArcGIS Online Dashboard that visualizes the data on a public web map.
Workspaces
The demo has been split into three key workspaces. One workspace gathers the data and sends it in an appropriate format for other workspaces to receive, and two workspaces process the data and take action.
1: Retrieving the Real-Time Data
The first workspace gathers the data from the source and processes it before sending it to other workspaces that will act on the real-time data. This first workspace acts as an intermediary between the source and our applications.
For demonstration, we used a pre-recorded dataset that streams data at 30-second intervals to the WebSocketSender. We used simulated real-time data because the demo must always be available on Safe.com, and the U.S. Geological Survey could be unreliable or have downtime.
This is usually not the case. Instead, you can connect directly to the source through a web connection and send the data out each time new data is received through a WebSocketSender.
Retrieving Data From the Source Using a REST API
The first step in our real-time data simulation workspace was retrieving data from the source API. We made one API call for each water gauge station, reading station IDs from a simple list.
@Value(site_num) is the Station ID and is used in the Request URL to get the current water levels for each station.
We then removed unneeded attributes using an AttributeManager. We used a Timestamper so we could sort the data chronologically, ensuring it streamed in the correct order. Finally, we packed the data into a JSON string and stored 24 hours of recorded data in an Excel sheet.
Sending Out Pre-Recorded Data
We created a second workspace that sends this data out in 30-second intervals through a WebSocketSender. We assigned the WebSocketSender a unique ID called “StationInfo”. We used this same unique ID in later workspaces to ensure that we connected to the correct WebSocket stream.
Many more Connectors can run in stream mode.
2: Creating an Email Notification Service
Rising water levels from rainfall, dam releases, or snowmelt often mean danger. Knowing what is considered a dangerous water level at each station, we can easily create an automated email notification service. This workspace runs in the background, constantly monitoring water levels and ready to alert in the event of a flood. This improves response time and means that as long as the system is up and running, you don’t need to worry about checking water levels yourself. This is made possible with FME’s TestFilter transformer and Emailer transformer.
Retrieving Values From the WebSocketReceiver
In the first workspace, we used a WebSocketReceiver to receive a JSON message from the WebSocketSender. The JSONFlattener then extracts the WebSocket message into attributes and values.
Limiting the Frequency of Emails
Emails in this demo are limited to once every 30 minutes so the receiving email address doesn’t get spammed. In a regular deployment, emails can be configured to send with every new flooding event. Values are sorted by station, and a sampler releases the last value received for each station when the time window ends.
Testing Current Water Levels for a Flood
We then set up a TestFilter to check water levels against known flood levels at each station. If a flood is detected, we send an email with a list of affected stations, the current water level, and the known flood height. The email body was customized using the HTMLReportGenerator. The HTML content generated by the HTMLReportGenerator is linked to an Email External Action in FME Flow using the Automations Writer. Below is a sample email.
Using the Group By Parameter
Note the use of the Group By parameter here. We grouped water level values by measurement time so that when features reach the HTMLReportGenerator, they are batched into one table and one email, rather than creating individual emails for each feature received.
3: Updating an ArcGIS Online Feature Layer with Real-Time Data
Real-time data can also be used to create visuals such as maps and dashboards, but to do this. We need a way to connect the real-time data to the map. To do this, we need to continuously upload the real-time data coming from our source to an ArcGIS Online feature layer.
Retrieving the Real-Time Data
In this step, the JSONFlattener connects to a “WebSocket message received” trigger in FME Flow and receives a JSON message from the WebSocketSender in the first workspace. We then extracted the JSON message into attributes and values.
Creating Geometry
Next, we added geometry to the data, allowing us to view it spatially on a map. We used a VertexCreator and a CoordinateSystemSetter to create a point object from the longitude and latitude attributes.
Filtering Attributes and Updating an ArcGIS Online Feature Layer
When updating an ArcGIS Online feature layer, the fields in the data you upload must match the fields and types in the ArcGIS Online feature layer. We filtered and formatted the attributes to match the fields in the ArcGIS Online feature layer. We then used the ArcGIS Online Feature Service Writer to update an ArcGIS Online feature layer.
With the feature layer updated in real time, we created a dashboard in ArcGIS Online showing which stations are currently experiencing flood levels. You can view the dashboard here
Tips and Tricks for the ArcGIS Online Feature Service Writer
When using the ArcGIS Online Feature Service Writer, it is important to understand key parameters before moving forward. Real-time data is a unique challenge, so hopefully, this clarifies any difficulties you run into.
Please note that if you use version 2024.0+, you must download the Esri ArcGIS Connector package from FME Hub and update your web connections. For more information, please see Working with Esri ArcGIS Feature Services in FME
Tip #1: Set the writer to read 1 feature per request
In the Navigator > AGOL Feature Service Writer > Parameters > Advanced, there is an option to set Features Per Request. Set this to 1. This ensures that when the ArcGIS Online Feature Service Writer receives a feature, it will write it out immediately. If this value is greater than 1, the writer will not update data in real time.
Tip #2: Make sure you have OBJECTIDs attached to your data
ArcGIS Online uses OBJECTIDs to identify unique features when used in UPDATE mode. If no OBJECTIDs are attached to the data we are sending out, the writer will not successfully update the data. We can add OBJECTIDs by merging a table with each feature and its OBJECTID into the data we read through the WebSocketReceiver. This is accomplished with a FeatureMerger.
Enabling the Workspaces in FME Flow
Finally, we used FME Flow to automate the workspaces and run them indefinitely.
We ran the first workspace, which simulates real-time data, using the Run Workspace function. Because we wanted to run the workspace in a loop, we enabled the Run Until Canceled option. Run Until Canceled restarts the workspace after it reaches the end of the pre-recorded dataset. We then ran the other two workspaces in separate automations.
Email Notification Service Workspace
We built an automation that connects the automation writers' output to an external email action. We used a manual trigger to run the workspace as if using the Run Workspace function in FME Flow. This lets the workspace collect data from the WebSocketReceiver and send bulk emails. With the email action connected, it sends an email containing a list of stations that triggered a flood alert.
ArcGIS Online Feature Layer Updater Workspace
We built an automation that lets the ArcGIS Online feature layer updater workspace access the WebSocket message sent by the first workspace. As a result, the ArcGIS Online feature layer updates continuously with real-time water level data.
Connecting it all with WebSockets
As mentioned earlier in the article, WebSockets allow data to flow freely between servers and applications. To enable the WebSocket connection, we created a publication with a Stream ID that matches the one used in WebSocketSender and WebSocketReceiver in our workspaces.
Conclusion
This is just one of the many practical uses of real-time data. In a flood monitoring system, we could integrate far more complex applications into this use case. For example, we could create a work order if water levels breach a designated flood height, dispatching a unit to deliver sandbags and other flood mitigation strategies. Simple tasks like these, which used to require human intervention, can now be automated with peace of mind using Streams in FME Flow.
For step-by-step instructions on how to create this workspace yourself, please see the two-part tutorial Real-Time Situational Awareness (Flood Monitoring) - Part 1, which creates the feature layer and sets up a workspace to update it, and Real-Time Situational Awareness (Flood Monitoring) - Part 2, which will take the workspace onto FME Flow to automate the process.