Real-Time Situational Awareness (Flood Monitoring) - Part 2

Introduction

This article is a continuation of the Real-Time Situational Awareness (Flood Monitoring) article. This article outlines how you can use the FME platform to take action on real-time data with automation.
 

Step-by-step Instrctions 

Part 4a: Creating an Email Notification Service

With real-time data constantly being streamed into FME, we can take advantage of this with a variety of tools and transformers. With known flood levels at each station we can test the current water levels against these flood levels and send out an email in the case of a flood event.

1. Start a new Workspace
Open up a new blank workspace in FME Workbench.

2. Add a Creator
First we need to add a Creator to the workspace, allowing us to initialize our WebSocket connection. Open up the Creator and set Create at End to Yes. 

3. Add a WebSocketReceiver
Next we need to connect to the WebSocket message being streamed out by the simulated real-time data workspace. Add a WebSocketReceiver to your workspace and connect the Creator to it. Open up the WebSocketReceiver and set the Parameters as follows:

• WebSocketServer URL: 
  • If you are using FME Server on your local machine: ws://localhost:7078
  • If you are using FME Server on a Server or Cloud instance: wss://<your server URL here>:7078/
• Verify SSL Certificates: No
• Connection Preamble: 

{

   ws_op: "open",
    ws_stream_id: "StationInfo" 
}

• Output Attribute: IncomingMessage

4. Add a JSONFlattener
The WebSocket connection being sent out by the simulated real-time data workspace will be received in the form of a JSON message. The JSONFlattener allows us to extract the data from the JSON message into attributes. Add a JSONFlattener to the workspace and connect the WebSocketReceiver to it. Open up the JSONFlattener and set the JSON Document to the Attribute Value “IncomingMessage”.



Next, we need to set up the Attributes to Expose parameter for the JSONFlattener. Open up the JSONFlattener and click the three ellipses next to Attributes to Expose. Add the following attributes:

• SiteName
• GageHeight
• timestamp

Click OK and OK again to exit the dialog.

5. Add an Excel Reader
We need to read in an Excel sheet containing the Flood Stage values as determined by the U.S. Geological Survey at each location. Add a reader to the workspace, set the Format to Microsoft Excel and select the Nashville_FloodStages.xlsx file as the Dataset. Keep all the other parameters as default. Press OK to add the reader to the workspace.

6. Add a FeatureMerger
We’re going to merge the features coming in from the WebSocket with the Nashville_FloodStages Excel sheet so that we can later compare the current water levels with known flood levels. To do this we need to add a FeatureMerger to the workspace. Connect the Excel reader feature type to the Supplier port and the JSONFlattener to the Requestor port. 

Open the FeatureMerger parameters. Under Transformer Mode, set Suppliers First to Yes. Under Join On, set Requestor to SiteName, and Supplier to Site Name, leaving the Comparison Mode to Automatic. Accept all other default parameters and press OK to close the window. 

7. Limit the Frequency of Emails
To limit the amount of emails being sent out and to prevent spam to your inbox, we need to add three transformers that will limit the email frequency.

First, add a TimeWindower and connect the FeatureMerger Merged port to it. Set the Window Duration to the value of your choice; we’ll use 5 minutes here. Set Time Window Starts to With First Feature and change ID Type to Window Number. Click OK to exit the dialog.


Secondly, add a Sorter after the TimeWindower. Open the Sorter and turn on Group Processing, setting it to Group By _window_id and set Group By Mode to Process When Group Changes (Advanced). Under Sort By, add the following:
 

Finally, add a Sampler and connect the Sorter to it. Open the Sampler and turn on Group Processing, setting it to Group By Site Name and _window_id. Set Group By Mode to Process When Group changes (Advanced). Under General, set the Sampling Rate (N) to 1, and the Sampling Type to Last N Features.


8. Add a TestFilter
Next we can add a TestFilter to test the current water levels against the known flood levels. Add a TestFilter to the workspace and connect the Sampler to it. We must set the following Tests to filter the incoming water level data.
 

9. Add an AttributeCreator
Next, we can send all the resulting passed test values to an AttributeCreator before sending the results to an HTMLReportGenerator. Add an AttributeCreator to the canvas. Open it up and set New Attribute to message, and Attribute Value to Action Stage. Connect the FeatureMerger Action Stage ALERT output port to the AttributeCreator. 

The other TestFilter output ports also need an AttributeCreator. Copy and paste the AttributeCreator three times, and attach each Alert output port to its own AttributeCreator. Open up each AttributeCreator and change the Attribute Value to the corresponding port it’s attached to. For example, the AttributeCreator connected to the Flood Stage ALERT port will have the Attribute Value of Flood Stage.


10. Add an HTMLReportGenerator
In order to create a table of all the Stations that passed the individual tests, we can use an HTMLReportGenerator. The HTMLReportGenerator lets you include HTML formatting like headers or tables in your emails.

Add an HTMLReportGenerator to the workspace and connect all of the AttributeCreators to it. Open up the transformer and enable Group Processing. Set Group By to timestamp & message.


This means that values with the same timestamp and flood stage breached will be processed at the same time, allowing us to include multiple stations into one HTML report if they all passed the same test. We use Group By (Process When Group Changes) here as opposed to Group By (Process at End) because the workspace will run indefinitely. Group By (Process When Group Changes) allows the workspace to constantly move values to the emailer whereas Group By (Process at End) would cause the workspace to never send an email since there isn’t an end to the workspace. 

We are going to create five different sections for this HTML Report. To create a section, click the first row under Page Contents and choose the Page Content you want to include. Then, add the rest of the contents in subsequent rows. Note that after choosing a Page Content type, you have to click somewhere in the transformer dialog to confirm your selection and see the parameters to fill in under Content Settings,

• Custom HTML:

  • Set the Custom HTML to “You're receiving this email because the stations below have breached the <b>@Value(message)</b> flood alert.”

• Separator

  • Use the default Separator Thickness of 1

• Table

  • Set Table Style to Bordered

  • Set the Column Settings as follows:

• Separator

  • Use the default Separator Thickness of 1

• Custom HTML

  • Set the Custom HTML to “<br><i>This is an automated message sent by the Nashville flood warning system</i>”

Click OK.

11. Sending Out an Email
Finally we can send out an email using the FME Server Automations writer which will enable us to send the result of the HTMLReportGenerator as an email using FME Server Automations. 

Add an FME Server Automations writer to the workspace and set the Feature Type Definition to Automatic and press OK. In the Feature Type dialog, set Feature Type Name to Flood Stage Email Alert. Connect the writer feature type to the HTMLReportGenerator, and double-click the writer feature type. Go to the User Attributes Tab, select Manual, and then remove the timestamp attribute so you are only left with html_content. 

Click OK.

12. Publish Workspace to FME Server
Save the workspace as Email_notification.fmw. We now need to upload this workspace to FME Server. Click the Publish button in the toolbar at the top of FME Desktop. Select your FME Server Connection and press next. Assign the workspace  to the Flood Notification Demo Repository and name it Email_notfication.fmw. We need to upload the flood values Excel sheet alongside the workspace, so press the Select Files button in the bottom right.

In the window that opens, check off the file that is listed and then press Select Location... 


Choose Upload to a shared resource folder. Use Data/FloodDemo and then press OK. Select the Nashville_FloodStages.xlsx file, then press OK again to close the file upload window and then press Next. Leave the Register Services window with the default values and click Publish. 
 

Part 4b: Running the Email Notification Service in FME Server

Now that we have a workspace that can test incoming water level values for a flood, we can connect the Email_notfication.fmw workspace to the Emailer action in FME Server to send out an email every time a flood is detected.

1. Create an Automation for the Email Notification Workspace
Go to your FME Server home page and click on Automations to open the drop-down menu, and then click on Build Automation.

2. Add a Manual Trigger
In the Automation creator, click on the Trigger to open the parameters. In the parameter window on the right, set Trigger to “Manually trigger event (Max 1 per Canvas)” and un-check Prompt for JSON on Trigger.

3. Add a Run Workspace Action
Next we need to add the workspace that will be run by the Manual Trigger. Add an Action to the canvas, then open the action parameters.. Select Run a Workspace as the action and the Email_notification.fmw workspace as the workspace. Ensure the Source Microsoft Excel File(s) parameter is set to use your uploaded data. It should look something like $(FME_SHAREDRESOURCE_DATA)/FloodDemo/Nashville_FloodStages.xlsx. Click Apply.

4. Add Email External Action
We need to add Email External Actions to each of the output ports from our Run Workspace Action. To do this, click on the orange plus button in the bottom left corner, select the blue paper plane symbol, and drag it onto the canvas. Connect the Action Stage Email output port to it. 

5. Set up the Email External Action
Click on the External Action square to open up its parameters. Set the Action to “Send an email”. Click on the blue “Load Template” button and choose an email service. Set the parameters as follows:

• SMTP Account (optional): <your sender email address>
• SMTP Password (optional): <your sender email address password>
• Email To: <the account you want to email the alert to>
• Email From: <your sender email address>
• Email Subject: Flood Alert
• Email Format: HTML
• Email Body (optional): html_content 
  • To get this, click the drop-down arrow, click on Workspace, click on Flood Stage Email Alert, and then choose html_content

Once you’re done, click the Validate button at the bottom left to test your email connection. If everything is valid, then click Apply.


6. Starting the Automation
Save your automation as Flood Email Notification. You can now press the green Start Automation button to begin the Automation. The automation will start and a Trigger button will appear next to the red Stop Automation button. Click the Trigger button to begin the workspace. A green rectangle that says Automation Triggered should appear.


The workspace is now running indefinitely, and will send an email alert if water levels breach any of the flood tests. If you are having any issues with the Email External Action, consult the FME Server Troubleshooting: Email article.

You now have a flood monitoring system demo that updates an ArcGIS Online feature layer and sends out email alerts in the case of a flood!

7. End Streaming Workspace
Once you are done with the demo, in FME Server, go to Jobs > Running, then click on checkbox next to the Simulated_data.fmw workspace then click Cancel. 
 

Part 5 (Optional): Enabling Topic Monitoring for WebSockets

Topic monitoring is a good way to make sure that our WebSocket messages are being sent out by the Simulated_data.fmwt workspace. Go to your FME Server home page and click on Automations to open the drop-down menu, and then click on Notifications (Classic). 



Now click New to create a new publication. This publication will act as the link between the WebSocket and the workspaces. Set the parameters as follows:

• Name: StationInfo
• Protocol: WebSocket 
• Topics to Publish To: Click the plus icon and create a new topic called “StationInfo”
• Target URL: wss://<your server URL here>:7078
  • For example: wss://fme-demo.com:7078
• Stream ID: StationInfo

Once everything looks correct, press OK. Now click the Topic Monitoring header at the top of the page. Here we can tell FME Server to monitor for the topic we just created so that the WebSocket messages are able to flow freely through FME Server. Click the drop-down arrow beside the empty box and select StationInfo. You should now see the WebSocket messages being sent out by the Simulated_data.fmwt workspace. You can expect to see the following: 

Data Attribution

The data used here originates from data made available by the U.S. Geological Survey. For the purposes of this demonstration, data has been recorded, modified, and simulated as real-time data.