INSPIRE Flood Warning Assistant Tutorial Exercise 1: Data Extraction

Introduction

As with any project, we need to determine what data we are going to work with and where that data is going to come from. For the purposes of this exercise, we are going to make use of the following datasets:

• Finnish Flood Zone Data (WFS3).
• Basemap info from INSPIRE Geoportal: UK OS Open maps - metadata.
• Basemap info from UK OS Open maps - downloads - UK OS GML, Geopackage etc.
• UK EA Flood Monitoring Service.

Throughout this exercise, we are going to set up a couple of different workspaces that are going to allow us to begin to use the above datasets, and export them to a singular format that we can make use of in our Web Application.

Example 1: Finland Flood Zones

Part 1: Retrieve Data

For this portion of the exercise, we are going to start with a starting workspace, so open ‘WFS3readerFinlandNLS2geojson_v2_starting.fmw’ in FME Workbench.

You will see here we have a starting workspace that should look like this. What this workspace is accomplishing for now, is that because FME doesn’t currently have a pre-packaged WFS3 reader, we have to set up a workaround that accomplishes this using FME transformers. This will hit the WFS3 Server, and in the first part(that you can see here) we will write out the list of the layer names from that WFS3 to a CSV file. From there we have ingested the WFS3 Server layer names to allow the second part that we are going to set up already know which layers are present in the WFS3.

To put it plainly, this starting workspace is merely running the GetCapabilities URL of the WFS3 Server.

1. Create WFS Get Feature Request URL.
Add an AttributeCreator to the ‘Passed’ output port of the Tester. We are going to create a new attribute called wfsGetFeatureRequest with the following value:

@Value(href)?Limit=$(Limit)&bbox=$(Min_long),$(Min_lat),$(Max_long),$(Max_lat)

This is going to allow us to apply spatial filters to our queries for specific areas since the server limits query results to only 1000 features per request.

2. Prevent Server Overload.
Add a Decelerator after the AttributeCreator we just added. With the Processing Slowdown Method set to Per Feature Delay, and the Delay Per Feature (Seconds) to ‘2’.

3. View/Log the Requests.
Next, add an Inspector and a Logger to the canvas. Now we are going to connect these back to the AttributeCreator. We could manually connect these to the AttributeCreator or we could connect these to a Junction.

4. Adding a Junction.
To add a Junction, you can right-click on a connecting line (in this case it would be the AttributeCreator -> Decelerator line), and select Insert Junction. Alternatively, you can just type Junction on the canvas as if adding a transformer. Now connect the Inspector and Logger to the Junction.

5. GetFeature Request.
Add an HTTPCaller to the canvas, and connect it to the Decelerator. Open the parameters, set the Request URL to the wfsGetFeatureRequest attribute that we created in Step 1. Ensure the HTTP Method is GET.

6. Set Feature Type Name from WFS3 Response.
Now that we have requested the GetFeatures and received a response, we want to write out to a json file. Add an AttributeCreator, and connect it to the Output of the HTTPCaller. We are going to create an Attribute ‘_filename’ with the value - @Value(fme_feature_type)_layer.json
This will allow FME to create a JSON file for each Feature Type that is found from the WFS3 Response.

7. Rename the WFS3 Response attribute.
In order to ensure that we write out the correct GEOJSON response to the text file, which will be used again in another workspace. We need to change the response feature to a name the TextFile relies on. As such add an AttributeManager to the AttributeCreator we just added. Here you will rename the _response_body output attribute to ‘text_line_data’. This is to ensure the Textfile Writer will write the correct features.

8. Textfile Writer.
Now that the HTTP Response has been configured correctly, we can add a Textfile Writer. Once added to the canvas open the Textfile Writer parameters in the Navigator window and set the following parameters:

Destination Text File: results
Fanout Dataset(ensure option is toggled) with the Fanout Expression set to: _filename

9. Save the Workspace.
Save the workspace as ‘WFS3readerFinlandNLS2geojson_v2.fmw’ and Run this workspace.

After the first translation run, you should have a tulvaalue_layer.json created in the result folder.

10. Rename Workspace.
The WFS3 Server we are using only allows for 1000 features to be requested at a time, as such we need to run the workspace again. However, in order to prevent our results from being overwritten navigate to the results folder, and rename the json file we created, tulvaalue_layer.json, to tulvaalue_layerS1.json.

Now run the workspace again but this time change following parameters at run time:

Minimum Latitude: 63
Maximum Latitude: 65

and you should have another tulvaalue_layer.json created in the results folder. Rename this to tulvaalue_layerS2.json. Now save the workspace and Open a new blank workspace.

Part 2: GeoJSON to GeoPackage

At the beginning of the workspace we established that we are going to bring in our different datasets and export them to a single destination, a Geopackage database. This step of the WFS3 process will involve actually loading the WFS3 features gathered from the GetFeature Workspace into the Geopackage.

1. Add a Reader.
Open the 'Finnish_FloodZones_json2geopackage_starting.fmw' workspace in FME Workbench. Once opened, add a GeoJSON Reader by either typing GeoJSON or going to the reader menu. Click the drop-down arrow located to the right of the Dataset field, and select ‘Select Multiple Files/Folders’. When the Select File browser appears, select the “Add Files” button and navigate to the two json files we created from the previous workspace.

When returning to the Add Reader dialogue, ensure the Workflow Options is set as ‘Single Merged Feature Type’. This will create a single reader feature type for both files as opposed to a feature type for each file.

2. Flood metadata clean up.
Add an AttributeCreator following the Reader with the following New Attributes and values:

3. Date clean up and formatting.
Add a DateTimeConverter to the AttributeCreator. The format of the date attribute needs to be altered by removing the ‘-’.

• Datetime Attribute: _dateInput
• Format: %Y-%m-%d$
• Output Format: FME

4. DateTime Assignment.
We have reformatted the date, but now we also want to ensure we have the Hour and Minutes of the date, so we will create a new attribute to handle this new Date format. Using a pre-configured custom Transformer ‘DateHHMMstamper’, located in the Custom Transformers bookmark, we can create this attribute without the seconds. The reason being is we don’t need to worry about the seconds of a potential flood report. Connect this up to the DateTimeConverter

5. Feature Type and File Name creation.
Add an AttributeCreator, with the following parameters:

6. Geopackage Writing.
Add a Geopackage Writer. Feature Type name will be ‘floods’ and we will write to the pastFloods.gpkg file.
In the parameters, ensure Overwrite Database = Yes.

Past floods Geopackage result:

7. Run Workspace.
Save the workspace as, Finnish_FloodZones_json2geopackage.fmw and run the workspace.
 

Example 2: UK Flood Zones

Part 1: Infrastructure

1. UK OS Open Map Local GML - Infrastructure Data.
In a new workspace, add a ‘GML’ Reader. Select the drop-down arrow to the right of the dataset field, and select ‘Select Multiple Folders/Files’. Click Add Files, and select the following files:

Click OK.

When the Select Feature Type dialog appears, select the following:

• CarChargingPoint
• ElectrictyTransmissionLine
• FunctionalSite
• ImportantBuilding
• RailwayStation
• RailwayTunnel

Note that when using the generic GML reader, FME typically needs an application schema to interpret the GML. In the case of the UK OS Open datasets, the GML has an xsi:schemaLocation defined with a valid URL to the application schema. As long as you are online, FME can retrieve this. However, if you are not online or are behind a firewall, you may need to use the local application schemas which we included in the same folder as the source data. The same applies for the Open Roads GML data below.

When the Feature Types have been added to the canvas, we need to expose the fme_feature_type attribute for use in the workspace. As such, open one of the Feature Type Parameters.

Toggle the Format Attributes Tab, and toggle fme_feature_type. Click 'Apply to..' and ensure the following parameters are set:
'Feature Types to change' should have all Feature types checked by default. For the 'Format Attributes to Expose' ensure the fme_feature_type, gml_id, gml_parent_id, and the gml_parent_property are all selected. Click OK.

This should expose the attributes for all Feature Types.

Functional Sites Source Data in Data Inspector

2. Schema Mapping.
Add an AttributeCreator to the workspace. Connect this to each of the reader feature types we added in the previous step. We will now create two new attributes to be used in unifying the features to be written to our Infrastructure Geopackage:

3. Reprojecting Data.
With dealing with a number of different feature types, we want to ensure that the data is in a LL coordinate system. As such, attach a CSMapReprojector to the AttributeCreator. Set the Destination Coordinate System to LL84.

4. Schema Consolidation.
There is some attribution that is not necessary for our Flood Mapping demo, as such we can remove some of the attributes. In this case, we will add an AttributeKeeper. Keep the following attributes:

• siteTheme
• buildingTheme
• source
• infrastructureType
• distinctiveName
• featureCode
• classification
• gml_id
• type

5. Write to Infrastructure Geopackage.
Add a Geopackage writer. The Destination file will be ‘infrastructure.gpkg’, and in the Parameters, ensure Overwrite Database set to Yes, and the Coordinate System set to LL84. The User Attributes should be as follows for the writer.

6. Save Workspace.
Save Workspace as ‘os_OpenMapLocal2geopackage.fmw’

Part 2: Roads and Junctions

1. UK OS OpenRoads GML Data.
Add a GML reader to the blank canvas. Select the drop-down arrow to the right of the Dataset field and click ‘Select Multiple Folders/Files’.

From here click ‘Add Files’, and select the following files:

Click Ok, and followed by clicking OK a second time. This will bring up the Select Feature Types Window. For the purposes of this exercise. We only care about the road_MotorwayJunction and road_RoadLink Feature Types.

2. Feature Filtering.
As with any dataset, there are features within the road_RoadLink table that are not needed. As such, filtering these features makes the most sense. To do this, add a Tester transformer to the road_RoadLink feature type. Here a simple test can be orchestrated to remove certain features that do not match the criteria. For this exercise, filter out the Primary or Trunk Roads.

primaryRoute = true OR trunkRoad = true

3. Road Name Classification.
Following the Tester, add an AttributeCreator. A possible new road name will be created by concatenating some values together, namely: roadClassificationNumber, name1, and name2.

4. Road condensing.
With creating a new Road Name, we can now aggregate these features together to drastically reduce the number of features to work with. Add an Aggregator to the AttributeCreator, and for the parameters set the Group By: _roadName.

5. Schema Consolidation.
Attach an AttributeRemover this will allow the condensing or removal of excess attribution that is not required for our Destination Feature Types. The attributes we will remove are the List Attributes (feel free to copy this list and paste it into the ‘Lists to Remove’ parameter):

inNetwork{} inNetwork{}.gml_remoteSchema inNetwork{}.nilReason inNetwork{}.owns inNetwork{}.xlink_href inNetwork{}.xlink_title inNetwork{}.xsi_nil

An AttributeKeeper will be required for the road_MotorwayJunction feature, so add one there as well. Far less attribution is required for the Junction features we will write, so only the following attributes will be kept:

• gml_id
• junctionNumber
• gml_parent_id

6. Geopackage Writer.
Add an OGC Geopackage writer to the workspace. For the Dataset, navigate to the ‘\exercise1\results\majorRoads.gpkg’ dataset and select this as the Destination Dataset. Click parameters and toggle the Overwrite Database option. Click OK. Before we finish adding the Writer, ensure the Add Feature Type(s) is set to Import from Dataset.

The Import From Dataset will produce another Reader/Writer dialog, click the Parameters. Then Select the (...) located to the right of Table List: and select junctions and roads as the tables. Click OK twice. This will add the two feature types to your workspace. Link up the roads to the AttributeKeeper that is connected to the Aggregator. Connect the junctions to the AttributeKeeper_2.

7. Schema Mapping.
Reading from a GML based format, and writing to Geopackage will display some differences in schema, namely, ‘.’ are not supported in attribute names in Geopackages. As such, some schema mapping needs to occur to correct the attribute names. There are two options to achieve this, on the connection between the AttributeKeeper and roads, if you right-click > Auto Connect Attributes, you will see the attribute names be carried over to their prospective writer counterpart. However, if you wish to see what is actually occurring, if you right-click that same connection line, and select ‘Replace Link with AttributeManager’.

8. Save Workspace.
Save workspace as ‘os_OpenRoads2geopackage.fmw’. Run the workspace.

Junction and Roads Geopackage