Real-Time Situational Awareness (Flood Monitoring) - Part 1

Introduction

With the FME Platform, it’s easy to turn real-time data into actionable products that can save your organization time and money. FME Server allows us to connect that real-time data to various applications, such as a real-time updating dashboard or an email alert service. 

This article emphasizes the strengths of real-time data, and how we can use the TestFilter transformer to perform tests on the data coming in to take action on it. Additionally, it also highlights the usage of WebSockets to connect real-time data to other applications. 

For the purpose of this demonstration, data was pre-recorded and simulated. This how-to article follows the same processes. The workspace that simulates a real-time data stream has been provided to you with instructions on how to use it.
 

Step-by-step Instructions

Part 1: Creating an ArcGIS Online feature layer

We first need to create an ArcGIS Online feature layer before we can continuously update it with real-time data. This is done using FME Workbench and the ArcGIS Online Feature Service Writer.

1. Create a New Workspace
Start up FME Workbench and create a new workspace. 

2. Add a Microsoft Excel Reader
To create the feature layer in ArcGIS Online, we will use an Excel file of one recording at each station. This will create all the fields needed to update the feature layer with real-time data in the further steps.

Add a Reader to the workspace, set the Format to Microsoft Excel and select the provided Excel file AGOL_create.xlsx as the Dataset. Set the Coord. System to EPSG:4326. This will create points for each of the station locations automatically using the Longitude and Latitude parameters in the Excel file. Keep all the other parameters as default.
 
3. Add an ArcGIS Online Feature Service Writer
This step involves setting up an ArcGIS Online connection so that we can create a new feature layer. 

Add an ArcGIS Online Feature Service Writer to the workspace by typing “AGOL Feature Service Writer” on the canvas. In the window that comes up, press Parameters. 


In the Parameters window that opens up, press the drop-down arrow for ArcGIS Online Connection and select Add Web Connection. Choose a Connection Name of your choice and then click authenticate. From here on sign-in to your ArcGIS Online account. When you’re done, the window will close automatically. Back at the Parameters window, we need to set a name for the Feature Service. In the box, type “FloodDemo”. If everything looks like the image below then you can close the window by pressing OK. 


In the add writer window, make sure to set Layer Definition to Automatic, and then press OK to close the window. In the Feature Type dialog, set Layer Name to LiveData, and Geometry to arcgisonline_point and press OK. The writer feature type should now show up in the workspace. Connect it to the Excel reader feature type.

We can now open the writer feature type back up by double-clicking it. Click the User Attributes tab. Set Attribute Definition to Manual. Delete the Longitude and Latitude attributes by clicking each one and pressing the “-” button. Now, we need to set the data type for each attribute. Set the Type as follows:

• SiteName: String
• GageHeight: Double
• Action Stage (ft): Double
• Flood Stage (ft): Double
• Moderate Flood Stage (ft): Double
• Major Flood Stage (ft): Double

Your workspace should now look like this:


4. Run the Workspace and Verify the Results
Save the workspace as AGOL_Create.fmw and then run the workspace to create the new ArcGIS Online feature layer. Upon successful completion, the following should appear in the log window:


In ArcGIS Online, you should now see the new Feature Layer has been created, alongside a File Geodatabase file.


If you encounter any ArcGIS Online authorization errors, please consult this document: ArcGIS Online Web Connection Error: 401 Client Error: Unauthorized for url.
 

Part 2: Running the Simulated Real-Time Data Workspace in FME Server

We need to run the simulated real-time data workspace that sends data out as a WebSocket message to be received by the AGOL_update workspace. This workspace is included in the Files section of this article.
 

1. Publish Simulated_data Workspace to FME Server
Open the Simulated_data.fmwt workspace in FME Workbench, which you can download from the Files section on this article. Publish the workspace to FME Server Choose your FME Server Connection and press Next. Assign it to the Flood Notification Demo repository, and then name the workspace “Simulated_data.fmw”. Press Select Files… to upload the Excel file attached to this workspace. 

Check off the file shown and then press Select Location…  and then click Upload to a shared resource folder. Next click on the Data folder to open the drop-down menu, click New Folder, and call it FloodDemo. Click OK, then enable the Stream Measurement-1Day-Modified.xlsx,  then click Next and then Publish. 
 

2. Run Workspace in FME Server
Return to FME Server. On the sidebar in FME Server, click Run Workspace. 

Choose the Flood Notification Demo repository and then select the Simulated_data.fmw workspace. Leave the Service as Job Submitter and make sure that the Source Microsoft Excel File has appeared. It should say “$(FME_SHAREDRESOURCE_DATA)/FloodDemo/Stream Measurement-1Day-Modified.xlsx”.
Under Published Parameters, change the WebSocket Server URL to your Server URL if you’re using something other than ws://localhost:7078. 

Click on Advanced to reveal additional options. Enable Run Until Cancelled. With this option enabled, the workspace will re-run automatically once it has finished outputting all the data from the Excel file. 

Click the green Run button to start the workspace. Now that this simulated real-time data is being streamed out to your WebSocket with a unique ID, the other workspaces can use this data.

To check if your ArcGIS Online feature layer is being updated, you can go to the feature layer in ArcGIS Online, look at its Data and see when the data was last updated. (You may need to refresh the page).

Part 3a: Updating an ArcGIS Online Feature Service with Real-time Data

Now that we have created an ArcGIS Online feature layer, we can update the feature layer with new real-time data incoming from our source. In this case, the source is a simulated stream of real-time data, but in a real-world scenario, the data would come directly from a sensor. 

1. Create a New Workspace
Create a blank workspace in FME Workbench. If you have the workspace from Part 1 still open, save it before continuing. 

2. Add a Creator
First, we need to add a Creator to the workspace, allowing us to initialize our WebSocket connection. We can accept the default parameters.

3. Add a JSONFlattener
The WebSocket connection being sent out by the simulated real-time data workspace will be received in the form of a JSON message. The JSONFlattener allows us to extract the data from the JSON message into attributes. Add a JSONFlattener to the workspace and connect the Creator to it. 

4. Linking the WebSocket Message to the JSONFlattener
Since the workspace will be running in an automation on FME Server, we need to create a User Parameter that will redirect the incoming WebSocket message to the JSONFlattener. We can do this by opening the User Parameter Manager by right-clicking on User Parameters in the Navigator and selecting Manage User Parameters. 

Press the plus symbol to create a new Text parameter. Set the Parameter Identifier to “WebSocket” and change the Prompt to “WebSocket”. Leave all the other parameters as default. 

Press OK to close the window. 

5. Setting up JSONFlattener
Next, we need to set up the Attributes to Expose parameter for the JSON Flattener. Open up the JSONFlattener and set the JSON Document parameter to the WebSocket User Parameter just created. Next, click the three ellipses next to Attributes to Expose. Add the following attributes:

• SiteName
• SiteLocation.SiteXCoord
• SiteLocation.SiteYCoord
• GageHeight
• OBJECTID

6. Add a VertexCreator
We need to create a point object to be updated in the ArcGIS Online feature layer, so we can do this using a VertexCreator. Add the transformer to the workspace and connect the JSONFlattener to it. Open up the transformer and set X Value to the attribute value SiteLocation.SiteXCoord and set Y Value to the attribute value SiteLocation.SiteXCoord. 

7. Add a CoordinateSystemSetter
We also need to give this new point object a coordinate system. We can add a CoordinateSystemSetter to the workspace and connect the VertexCreator to it. Set the Coordinate System to EPSG:4326. 


8. Add an AttributeManager
Before we can write the data received from the WebSocketReceiver to the ArcGIS Online feature layer we need to filter out some attributes. Add an AttributeManager to the workspace and connect the CoordinateSystemSetter to it. Open up the AttributeManager and remove the following attributes:

• _creation_instance
• SiteLocation.SiteXCoord
• SiteLocation.SiteYCoord

9. Add ArcGIS Online Feature Service Writer
Now, we can update the ArcGIS Online feature layer we created in the previous part. Add an ArcGIS Online Feature Service Writer to the workspace by typing “AGOL Feature Service Writer” on the canvas. In the window that comes up, press Parameters. 


In the Parameters window, press the drop-down arrow for ArcGIS Online Connection and select the web connection you created in Part 1.

Click the three ellipses next to Feature Service and select the feature layer you want to write to. In this case we will choose the FloodDemo feature layer that was created in Part 1. Expand the Advanced parameters and set Features Per Request to 1. When you’re done, press OK to close the window. Set the Layer Definition to Automatic. Then, Press OK to close the window.


In the new window that shows up, set the following parameters:
Layer Name: LiveData
Geometry: arcgisonline_point
Writer Mode: UPDATE 

Then, click OK to close the window. 

Connect the AttributeManager to the ArcGIS Online Feature Service Writer.


We now need to set the Attribute Types to match the ArcGIS Online feature layer we created in Part 1. Open the LiveData writer feature type and go to the User Attributes tab. Set the Attribute Definition to Manual. Change the Types to as follows:

• SiteName: String
• GageHeight: Double
• OBJECTID: OID

Press OK to close the window when you are done. Save your workspace and name it AGOL_update.fmw.

10. Test Run the Workspace
We will test the workspace with a mock Excel file to make sure that our ArcGIS Online web connection is working. Add a reader to the workspace. In the reader, set the Format to Microsoft Excel, and set the Dataset to the AGOL_test.xlsx file provided. Set Coord. System to EPSG:4326. Press OK to add the reader. 

Connect the reader to the ArcGIS Online Feature Service writer feature type. Select the JSONFlattener and choose Disable. 


Now run the workspace. You should see in the log that the writer was able to successfully update the ArcGIS Online feature layer. The following should have shown up in your log: 


If you were able to successfully update the ArcGIS Online feature layer, then move onto the next step. If you encountered an error, go back through the previous steps and make sure that you did not miss a step.

A few things to check to troubleshoot any issues would be to make sure the writer is in Update mode and make sure that your ArcGIS Online web connection is valid.

Before we move onto the next step, right-click the Creator and choose Enable. Delete the AGOL_test.xlsx reader as we don’t need it anymore. Save your workspace. 

11. Publish Workspace to FME Server
We now need to publish this workspace to FME Server so that we can run it indefinitely, constantly updating the ArcGIS Online feature layer. Click the Publish button in the toolbar at the top of FME Desktop. Select your FME Server Connection and press Continue. Assign the workspace a new Repository called Flood Notification Demo and then name workspace “AGOL_update.fmw”. In the next screen, be sure to select the ArcGIS Online connection if it does not already exist and select Continue. Leave the Register Services window with the default values and click Publish. 
 

Part 3b: Running the ArcGIS Online Feature Layer Updater on FME Server

Next, we want to run the workspace in FME Server using an Automation. This will allow the workspace to run in the background, continuously updating the ArcGIS Online feature layer. 

1. Create an Automation for the ArcGIS Feature Layer Update Workspace
Go to your FME Server home page and click on Automations to open the drop-down menu, and then click on Build Automation. 


2. Set up the WebSocket Trigger
In the Automation creator, double-click on the “Trigger”. In the Trigger Details window on the right, click on the drop down menu and select WebSocket message received as the Trigger. Under Parameters set the Target URL to your Server URL if you’re using something other than ws://localhost:7078. (ex: wss://<your-server-url>:7078. FME Server uses 7078 as the default port for WebSocket Server requests. Ensure that port 7078 is open on your FME Server before proceeding. Next, set the Stream ID to StationInfo, then click Apply. Stream ID is the ID that was given to the real-time data within the Simulated_data.fmwt workspace. 


Click Apply.

3. Add a Run Workspace Action
Next we need to add the workspace that will be run by this trigger. Select the orange plus symbol in the bottom left corner and select the orange wrench labelled Action and drag it onto the canvas.


Connect the WebSocket to the Action by clicking and dragging the checkmark. 


Click on the orange Action square to open its parameters. Set the Action to Run a Workspace, then select the Repository and Workspace you published in Part 2 (AGOL_update.fmw). At the bottom under the Parameters box, select the drop-down arrow beside WebSocket, expand WebSocket, and select Message Content.


Click Apply to save the new settings. Name your automation AGOL_update and save it. Your automation should look like this:


You can now press the green Start Automation button to begin the automation. The AGOL_update.fmw workspace will now run every time a new WebSocketMessage is retrieved, updating the ArcGIS Online feature layer with real-time data.

You can find Part 2 of this article here. Part 2 outlines how we can use FME Server to automatically send out email alerts when a flood is detected.