INSPIRE Flood Warning Assistant Tutorial Exercise 2: Live Data Feed to Geopackage

Introduction

The second part of this Flood Warning Assistant requires a live data feed of current floods from the UK EA Flood Monitoring Service(listed below). We are going to pull this data into a Geopackage file labeled as Current Floods.

UK EA Flood Monitoring Service:
https://environment.data.gov.uk/flood-monitoring/id/floods?min-severity=4

Step-by-step Instructions

1. Opening Starting Workspace
Open ‘UKEA_FloodWarnings_RESTjson2geopackage_Starting.fmw’ in FME Workbench. This will be the starting workspace for this exercise. In this workspace, the live data feed from the UKEA has already been configured with an AttributeCreator and the FeatureReader.

2. Send the HTTP Request
First, start with requesting features from the URL we created in the AttributeCreator, so add an HTTPCaller and place it in the Log Request Feature bookmark. The Request URL will be the _request1 attribute and ensure it is a GET request.

3. JSON Writing
From the HTTPCaller output port, connect an AttributeFileWriter. This will allow us to write out the json response we receive from the HTTPCaller to a json file.

4. Log File
Following the AttributeFileWriter, no other transformer is needed for this section of the workflow, however, for record-keeping. Attach a Logger to the Output port; this will provide us with details when the workspace has been run.

5. Flood Features
From the FeatureReader’s Generic port, attach an AttributeExposer into the Attribute Exposer exposes hidden geometry bookmark. This will allow the json features’ attributes we read in, to be used within the workspace immediately. While the attributes would be visible when inspecting, Workbench would not be explicitly aware of them yet. As such, expose the geometry of the feature by exposing the ‘floodArea.polygon’ attribute.

6. Sample Features
Add a Sampler following the AttributeExposer. This is where the number of floods to show will be determined, this will be passed in by the user. Set the Sampling Type to ‘First N Features’. To set the Sampling Rate (N) as a User Parameter, select the drop-down arrow.

Fill the User Parameter definition as the following:

• Name: MaxFloods
• Prompt: Max Floods:
• Default Value: 999
• Published: Checked
• Optional: Unchecked

7. Create 0 Flood Attribute
Add a StringLengthCalculator and connect it to the Sampled port of the Sampler. This will be used to test the length of the floodArea.polygon attribute. The result attribute will be ‘_length’ this will allow the determination if no flood polygons were produced.

8. Test for No Floods
Following the StringLengthCalculator, add a Tester to the Result Features but No Geometry bookmark. This is where if no flood polygons exist, they will be filtered out. The test will be the following: _length > 0.

9. Prevent Flood Polygon Server Overload
From the Passed port of the Tester, attach a Decelerator and place it in the Get flood polygon associated with flood feature bookmark. This is to prevent the server from being overloaded. The following parameters will be used:

Processing Slowdown Method: Limit Features Per Second
Maximum Features Per Second: 2

10. Fetch the Flood Polygon Geometry
Attach an HTTPCaller to the Decelerator, here we will use the floodArea.polygon attribute as the GET Request URL. This will return the geojson of the Flood polygon.

11. Split the Flood Feature Geometry
From the Output of the HTTPCaller port, attach a JSONFragmenter. This will allow the splitting of the JSON response from the GET Request earlier. The JSON attribute will be the _response_body attribute from the HTTPCaller. The JSONFragmenter allows us to specify the JSON Query in order to go to the geometry that is nested deeper in the file. In this case, the JSON Query will be the following:

<["features"][*]["geometry"]

Ensure the Fragment as Format is set to JSON, and that the Flatten Query Result into Attributes is toggled to Yes.

12. Expose Required Attributes
As mentioned previously, certain transformers won’t populate the attributes to be used in FME Workbench (even if they are viewable when Inspecting the data). The JSONFragmenter is one transformer that does this. As such, from the Fragments port, attach an AttributeExposer and expose the following attributes:

• geometry
• floodAreaID
• description
• message
• severity
• severityLevel

13. Add Date
Add a DateTimeStamper to the AttributeExposer. The format of the date attribute that we are going to add needs to be in UTC.

• Type: DateTime
• Zone: UTC
• Result Attribute: _date

14. DateTime Assignment.
From the DateTimeStamper, a Custom Transformer will be added, the DateHHMMStamper, this is similar to the DateTimeStamper, however, we have used it to create a custom DateTime Stamp, that does not contain seconds.

15. Feature Type and File Name Creation
Add an AttributeCreator to the Output of the DateHHMMStamper, with the following parameters:

16. Replace Geometry of the Feature with the GeoJSON Response
From the AttributeCreator, attach a GeometryReplacer. This will be used to replace the Geometry of the flood feature. Ensure the Geometry Encoding is set to GeoJSON, and the Geometry Source is the _response_body attribute. Then set Remove Attribute to No.

17. Set the Coordinate System
Add a CoordinateSystemSetter to the Output port of the GeometryReplacer. Set the Coordinate System to EPSG:4326.

18. Connect to Writer
Connect the Output of the CoordinateSystemSetter up to the floods Feature Type of the Geopackage Writer that is already set up.
 

19. Run Workspace
Save the workspace as 'UKEA_FloodWarnings_RESTjson2geopackage.fmw' and Run Workspace.
 

Optional: If no Flood Geometry or/and Error Handling

In Steps 7 and 8, there was a test to check if there was No flood geometry. Some additional logic could be added here to provide the user with additional information.

19. No Flood Information
From the Tester located in the Result Features but no geometry bookmark, attach an AttributeCreator to the Failed port. It is here, some information can be created to provide the user:

20. No Feature Tester
As mentioned before, if no feature is produced some information should be provided to the user. As such, hook up the NoFeaturesTester custom transformer to the Generic port of the FeatureReader. Place the NoFeaturesTester into the Handle no result features bookmark.

21. No Flood Information
From the NoInput port, copy the AttributeCreator created in Step 19. This will provide the user with the same information outlined above.

22. Log No Features Information
Add a Logger and attach both AttributeCreators, from Step 19 and 21, to the Logger.

23. Save Workspace
Save Workspace as ‘UKEA_FloodWarnings_RESTjson2geopackage.fmw’

24. Run Workspace.

Results:

Completed Workspace