Flood Notification Scenario

Introduction

In this scenario we create an automated alerting framework that will phone customers warning them of a potential flood. In this scenario, an insurance company has a database table containing all of their customers. We create a workflow that takes the customer’s address, geocodes it and then compares it to the potential flood area. If the customer lies within the flood zone we use Twilio to send an automated SMS warning them.
 

Step-by-step Instructions

Part 1: Retrieve Customer Data

1. Read customers from API

The customers are accessible via an API hosted on AWS:

https://h9nht9dzsl.execute-api.us-west-2.amazonaws.com/dev/api

Explore the API documentation. The API offers four calls that allow you to create, read, update and delete (CRUD) customers from the database.

• Use the UI to retrieve all of the customers currently in the database (Hint: GET /customers). On doing this you will notice that the Request URL is: https://h9nht9dzsl.execute-api.us-west-2.amazonaws.com/dev/api/customers/
• Copy the URL and in a blank workspace, add a Creator transformer and connect it to an HTTPCaller transformer. The Creator will create a single point to trigger the translation.
• In the parameters of the HTTPCaller, set the Request URL (https://h9nht9dzsl.execute-api.us-west-2.amazonaws.com/dev/api/customers/) and set the HTTP Method to GET

2. JSONFragmenter

Add a JSONFragmenter transformer, this transformer extracts portions of JSON text into features. In the parameters, set:

• JSON Attribute: _response_body
• JSON Query: json[*].
• Flatten Query Results into Attributes: Yes
• Recursively Flatten Object/Arrays: Yes

If you look at the API in a web browser again, you can see that there are six attributes.

Click on the ellipsis for Attributes to Expose to bring up the Enter Values dialog box. On a separate line for each enter: name, phone, address, customer_id, email and registered_for_alerts. Ensure your spelling is correct so it matches exactly with the API.

3. AttributeKeeper

We are only interested in the attributes we just added in the previous steps (name, phone, address, customer_id, email, and registered_for_alerts), so we will add an AttributeKeeper to only keep those attributes.

4. Add a Geocoder

The API only has addresses, there isn’t any geometry associated with it, so we will have to run the addresses through a geocoder.

• Add a Geocoder transformer, open up the parameters and set the following parameters:
  • Geocoding Service: Bing
  • API Key: Copy the API key from the READ_ME provided.
  • Mode: Forward
  • Street Address: address attribute.
• Exit out of the parameters and save your workspace.
• Connect Inspector transformers to both the Output and <Rejected> ports on the Geocoder transformer and run the translation. Note, you will need to change your Rejected Feature Handling Default to Continue Translation (Tools >> FME Options >> Translation).
• In Data Inspector, you should see points, click on one of the points to confirm the correct data is under each attribute heading. You can add a background map within the Data Inspector. 

Part 2: Generate a Flood Polygon

5. Read DEM and reproject

• Add a Canadian Digital Elevation Data (CDED) Reader to read the DEM-Clipped.dem file.
  • On FMEUC VM: C:\FMEData2017\Data\ElevationModel\DEM-Clipped.dem
• The DEM currently doesn’t have a coordinate system so we will need to reproject it. Add a CsmapReprojector and set the Destination Coordinate System to EPSG:4326

6. Add NOAA Flood Feed

We will now retrieve from tide and current data from the National Oceanic and Atmospheric Administration (NOAA), this data is based out of the United States, but Vancouver is reasonably close to the border, so for this example, we will call data from Cherry Point, Washington.

You will notice in the URL above there is a ? in the URL, and then after that there is a list of key/value pairs. These are called query string parameters and they are used to pass data to the web server. In the HTTPCaller you can either just paste the query string parameters in the Request URL box with the URL. Or you can set them in the Query String Parameters table in the Request section. The advantage of using the table is it makes things a bit more understandable and maintainable. We will do both.

• Add a Creator and HTTPCaller transformer to a blank space on the canvas.
• Set the following in the HTTPCaller:
  • Request URL: https://tidesandcurrents.noaa.gov/api/datagetter?begin_date=20170415%2010:00&end_date=20170415%2010:24?uct=water_level&datum=mllw&units=metric&time_zone=gmt?plication=web_services&format=json
  • HTTP Method: GET
  • Query String Parameters table:

  Name: station

  Value: 9449424

7. JSONFragmenter and JSON Tree

Connect a JSONFragmenter to the Output port on the HTTPCaller. Set the following:

• JSON Attribute: _response_body
• JSON Query: json["data"][*]
• Flatten Query Results into Attributes: Yes
• Recursively Flatten Object/Arrays: Yes
• Attributes to Expose: v (which is the water level)

Advanced

To generate the JSON Query (json["data"][*]), you can use a trick with the FeatureReader.

• Add a temporary FeatureReader to the canvas, set the format to JSON, and the dataset to the URL: (https://tidesandcurrents.noaa.gov/api/datagetter?begin_date=20170415%2010:00&end_date=20170415%2010:24&station=9449424?uct=water_level&datum=mllw&units=metric&time_zone=gmt?plication=web_services&format=json)
• Click the Parameters button under the Reader heading. Set the Schema Scan Mode to JSON Query. Highlight Query cell in the table, click … to browse to the tree. Select the data > <array values> node. The code appears in the Selected Item box. Copy the code, then delete the FeatureReader and paste the code into the JSONFragmenter

8. StatisticsCalculator

We will now calculate the summary statistics for the range of water levels. Add a StatisticsCalculator and set the Attributes to Analyze to v.
 

9. Climate Change Factor

We need to factor in for climate change in this scenario because we are using live APIs and it currently isn’t flooding in this region Let’s set up a User Parameter. In the Navigation Window, right click on User Parameters and click Add Parameter…, enter the following:

10. Merge DEM and second HTTPCaller branch

Now we will merge the DEM and the NOAA API together.

• Add a FeatureMerger transformer, and connect the Requestor Input port to the Output port on the CsmapReprojector
• Connect the Supplier Input port to the Summary Output port on the StatisticsCalculator.
• In the FeatureMerger, set both the Requestor and Supplier to the value 1 (this is acting as a constant and you could use any identical string).

Tip: Because there is only one feature coming into the Requester and Supplier ports, setting the value to 1 just merges the two features together.
 

11. AttributeCreator

To utilize the ClimateFactor parameter, we will add the climate factor value to the real water level. Add a AttributeCreator, connect it to the Merged Output port on the FeatureMerger. In the parameters set the following:

• New Attribute: flood_level
• Attribute Value - open up the arithmetic editor and enter the following:

@Value(_max) + $(ClimateFactor)

12. Create Flood Polygon

Next we need to simulate a flood and extract a polygon that represents the flooded area. To do this requires some relatively complex raster calculations. We have completed this for you to save time. If you are interested in how it works, have a look in the custom transformer and you can also check out this blog post or check out one of the many articles in the Knowledge Center

Add the FloodPolygonExtractor transformer (the custom transformer is installed on the virtual machines, but you can also download and install from the this article). Connect it to the Output port of the AttributeCreator transformer. Set the Flood Level parameter to be the flood_level attribute you just created.

Connect an Inspector transformer and run the workspace and have a look at the output. You should see a polygon representing the flood extent.
 

Part 3a: Determine Who is Affected

13. Join Parts 1 and 2

Now we will determine who is affected by the flood.

• Add a Clipper transformer to the canvas.
• Connect the Geocoder Output port to the Clippee Input port on the Clipper
• Connect the FloodPolygonExtractor Output port to the Clipper Input port. You may want to reorder the Clipper and Clippee ports to tidy up your workspace.
• Connect an Inspector to the Inside port on the Clipper transformer and run the workspace with prompt. Once prompted, set the Climate Change Factor high (9-12) for maximum results. Viewing the output in the Data Inspector you will see a few customers affected by the flood (results may vary depending on the day).

Part 3b: Determine Who is Affected - Create Customers

14. Create new customers

As you can see we don’t have a lot of customers that will be affected, even if the Climate Change Factor is set to 12. Also, within the customer attributes, no one has signed up to be alerted for floods (registered_for_alerts attribute). We will need to create some new customers for our scenario to be effective. To do this we will use the POST command in the API to add records to the database.

Open up a new workspace, add a Creator transformer and an AttributeCreator. Enter the following table into the AttributeCreator. For the values in (brackets) enter your own information or make something up, just ensure the address is a valid Vancouver, BC address.

15. Write to the API

There are two ways to write to the API. You can add a custom transformer we have created (simpler method), or, the advanced way is to add an HTTPCaller and configure it yourself. Choose either the Simpler or Advanced method.

Simpler Method

• Add the CreateNewCustomer custom transformer (custom transformer is installed on the virtual machines, you can also download and install from this article) to the canvas and connect to the AttributeCreator.
• Set the User Parameters on the custom transformer
• Then connect loggers to each Output port and Run the translation.
• Open the API explorer and make a GET request to the API to return all users. Once the JSON returns, use ctrl-f (command-f) to find your name within the API to confirm the workspace worked.

Advanced

Add an HTTPCaller and connect to the AttributeCreator. The POST HTTP method is comparable to an UPDATE in the database world. When you post data to a URL you will usually create a new object using the data you passed in. Set the following parameters on the HTTPCaller.

Then connect loggers to each Output port and run the translation. Open the API explorer and make a request to the API. Once the JSON returns, use ctrl-f (command-f) to find your name within the API to confirm the workspace worked.

Part 4: Twilio Custom Transformer

We will be using Twilio to alert the customers who have signed up for alerts when there is flooding. Twilio is a SaaS product that makes it really easy to send voice, video and SMS messages. They have created a really powerful set of APIs which abstract the complexity and allow developers to add communication capabilities to their apps (Lyft, Airbnb and Netflix all use it).
 

16. Test for customers who have signed up

Now that we have added our information to the database, we would like to receive a call message to our phone when a flood is imminent.

• Open the original FloodNotificationGenerator.fmw you were working on before.
• Add a Tester, connect it to the Inside Output port on the Clipper. In the Tester, set the following:
  • Pass Criteria: All Tests (AND)
  • Test Clauses:

  registered_for_alerts = true

  email=<your email>

The reason we are also setting the email on the Tester is because otherwise every time you ran your workspace, it would call everyone in the flood zone (everyone in the room).
 

17. Trigger Automated Call Using Twilio API

The final step is to use the Twilio API to make an automated call to all customers that are within the flood zone. There are two ways you can do this:

1. Simple Method: Download the TwilioCaller transformer from the FME Hub and configure it. This is detailed in step 18.
2. Advanced Method: Build the custom transformer yourself. This is detailed in step 19.

18. Configure TwilioCaller From FME Hub

• Download the TwilioCaller from FME Hub by using the quick type in FME Workbench (just start typing on the canvas)
• Attach it to the Passed port on the Tester transformer
• You can now jump to step 20.

Tip: Even if you don’t build the transformer, I’d suggest right clicking on it, clicking edit and looking at how it works. It interacts with the call defined here in the API documentation.
 

19. Build Twilio Custom Transformer - Advanced

Right click on the canvas and click Create Custom Transformer. This will open up a new window in Workbench. Set the following:

• Name: TwilioVoiceCaller
• Category: Web Services
• Overview: Uses the Twilio API to allow you to make outgoing calls from FME Workspaces. Twilio exposes a globally available cloud API that developers can interact with to build intelligent and complex communication apps.

19a. Setup Twilio Parameters

In the Navigator, right click on User Parameters and click Add Parameter. We will be doing this five times, enter each of the following tables into a new User Parameter.

19b. HTTPCaller to connect with Twilio API

Before we configure the HTTPCaller let’s take a look at the API documentation. This is fairly standard documentation and it shows we must POST several parameters to the API endpoint to make a phone call.

Add a Creator transformer and connect an HTTPCaller. Based on the API documentation see if you can configure the HTTPCaller yourself. The parameters need setting as following:

Request

Response

Authentication

19c. JSONFlattener

Add a JSONFlattener to the Output port of the HTTPCaller. In the parameters, set:

JSON Document: _response_body

Recursively Flatten Objects/Arrays: Yes

Attributes to Expose:

• sid
• date_created
• date_updated
• parent_call_sid
• account_sid
• to
• to_formatted
• from
• from_formatted
• phone_number_sid
• status
• start_time
• end_time
• duration
• price
• direction
• answered_by
• api_version
• annotation
• forwarded_from
• group_sid
• caller_name
• uri

19d. JSONExtractor

Error handling is a really important thing to think about when working with APIs. Often APIs will give you useful messages if you do something wrong. Workflows you build should capture these messages and display them to the user.

Add a JSONExtractor to the workspace and connect it to the Rejected Output port on the HTTPCaller. For the JSON Document: set it to the attribute _response_body

Then for Extract Queries, enter the following table:

19e. Create Input and Outputs

Finally, we will need to create input and outputs for our custom transformer.

Right click canvas and add click Insert Transformer Input. Connect the Input to the HTTPCaller.

Add an Output and connect it to the Output port on the JSONFlattener. Set the parameters to:

Attributes to Output: Specified Attributes Only

Uncheck:

• _headers{}.name
• _headers{}.value
• _http_status_code
• _response_body

Add another Output, rename it Rejected and connect it to the Evaluated Output port in the JSONExtractor. Set the parameters to:

Attributes to Output: Specified Attributes Only

Uncheck:

• _headers{}.name
• _headers{}.value
• _http_status_code
• _response_body

Back in the main workspace, make sure the TwilioVoiceCaller you just created is connected to the Tester transformer. The TwilioVoiceCaller is slightly different to the TwilioCaller on FMEHub. You can download a working TwilioVoiceCaller here.

20.  Configure and run Workspace

It is now time to run the workspace and alert everyone within the flood zone with an automated phone call. Before we do that, we need to configure the TwilioVoiceCaller (or TwilioCaller if you downloaded it from the FME Hub).

Use the following values. Make sure you go to the TwimlURL in a web browser and have a look at it. This is the XML that contains the words that will be spoken when the phone call is made.

Now it’s time to run the workspace. Note, this will call every user in the database that is within the flood zone. You might want to use the Tester to just filter out your user.
 

21. Wrap up

Save the workspace, which will also save the custom transformer. Switch back to the Main tab, which is the original workspace. Run with Prompt to test the workspace.