Converting from XML (Simple XML Reading Example)

Introduction

Most often, XML is read with FME using predefined XML profiles for specific XML formats or FME's XML reader with Feature Paths. Other approaches to reading include XfMaps, or FME’s Text File reader using XQuery and other XML processing transformers. These approaches are considered advanced and are covered in other articles. In this article, we will focus on feature paths.
 

XML Profiles

The first thing to check is whether or not the data you are trying to read is already supported in FME as a specific XML format or profile. For frequently used XML profiles, these are often implemented as their own FME format reader and writer. FME supports more than 40 different XML formats. Using these formats is always easier than trying to configure the generic XML reader/writer. Examples of XML formats supported in FME include: Open Street Map (OSM), Google KML, GPX, RSS, LandXML, and GML, to name a few. To see a complete list,  open up the reader or writer gallery and type XML in the search field.

Video

This video was recorded using an older version of FME. The user interface may be different, but the concepts are the same. 

Source Data

The source dataset is an RSS feed from DriveBC, where the information is contained within the Item element. 

<?xml version="1.0"?>
<rss xmlns:dbc="https://www.drivebc.ca/rss" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" version="2.0">
  <channel>
    <title>DriveBC Events</title>
    <link>https://www.drivebc.ca</link>
    <description>DriveBC Events</description>
    <ttl>1</ttl>
    <item>
      <title>Highway 17 - Both Directions - Current Planned</title>
      <link>https://www.drivebc.ca/~DBC-55614</link>
      <guid>https://www.drivebc.ca/~DBC-55614</guid>
      <description>Highway 17, in both directions. Bridge construction between Tannery Rd and Bridgeview Dr for 3.3 km (Surrey). Until Mon Sep 18 at 5:00 AM PDT. From 9:00 PM to 5:00 AM PDT daily. Full closures in both directions to erect steel girders for Pattullo Bridge Replacement Project.
Local Roads closed from 7PM to 5AM Sept 5,6,7,8,11,12 and 13th.
Watch for traffic pattern changes and detour signage. Next update time Mon Sep 18 at 5:00 AM PDT. Last updated Wed Sep 6 at 4:35 PM PDT. (DBC-55614)</description>
      <category>CONSTRUCTION</category>
      <pubDate>Wed, 06 Sep 2023 16:35:34 -0800</pubDate>
      <dbc:district>Lower Mainland District</dbc:district>
      <dbc:route>Highway 17</dbc:route>
      <dbc:type>Current Planned</dbc:type>
      <dbc:severity>Major</dbc:severity>
      <geo:lat>49.207272</geo:lat>
      <geo:long>-122.883444</geo:long>
    </item>
    <item>
      <title>Highway 17A - Southbound - Incident</title>
      <link>https://www.drivebc.ca/~DBC-54548</link>
      <guid>https://www.drivebc.ca/~DBC-54548</guid>
      <description>Highway 17A, southbound. Vehicle incident at the overpass of Highway 99. The southbound lane on the overpass is closed due to previous incident. Southbound traffic is being routed onto the northbound HOV / transit lane. Follow signs and watch for crews working. Next update time Mon Sep 25 at 12:00 PM PDT. Last updated Mon Sep 11 at 12:36 PM PDT. (DBC-54548)</description>
      <category>INCIDENT</category>
      <pubDate>Mon, 11 Sep 2023 12:36:08 -0800</pubDate>
      <dbc:district>Lower Mainland District</dbc:district>
      <dbc:route>Highway 17A</dbc:route>
      <dbc:type>Incident</dbc:type>
      <dbc:severity>Major</dbc:severity>
      <geo:lat>49.107669</geo:lat>
      <geo:long>-123.055643</geo:long>
    </item>
</channel>
</rss>

Step-by-Step Instructions

In the following exercise, we will see how to read XML elements as records and flatten their member elements into attributes. We will also see how to extract the geometries.
 
1. Open FME Workbench
Open FME Workbench and create a blank workspace. Click on the Reader button to add a new reader. In the reader parameters, type in XML for Format and select the XML (Extensible Markup Language). Then, for Dataset, paste in the following URL:

http://www.drivebc.ca/api/events/district/mainland?format=rss

The DriveBC API records all of the traffic and weather events throughout British Columbia, Canada. 

2. Open the Reader Parameters
Now click on the Parameters button, still inside the reader; these parameters will affect how FME reads in the XML file. Click on the ellipsis next to Elements to Match to open the dialog. 
 
Expand RSS by clicking on the arrow, then expand channel. Once channel is expanded, select item. Click OK three times. 


Note that we could also have just typed "item". The complete path is only important if the element type you are matching occurs at different levels in the dataset. We only selected the element we wanted to become a feature type. We didn’t check RSS or channel since this would create very large features - e.g. one feature per dataset or feature type.
 
3. View the Dataset
Before we can manipulate our data any further, we should look at the structure of the data. Since this is a live data stream, the data will appear different than what is in the following screenshots. 
Ensure Feature Caching is enabled, run the workspace, and then click on the Inspect Cached Features button. Optionally, you can connect an Inspector transformer and then run the workspace. 
 
When viewing the data, you will see various road incidents in British Columbia with latitude and longitude values. We can turn this data into points. 

Note: If you inspect the data and there are no current incidents being reported, download the provided dataset. 
 
4. Add a VertexCreator 
Add a VertexCreator transformer to the canvas and connect it to the item reader feature type. 
 
In the parameters, set the X Value to long and then the Y Value to lat, and then click OK. 


5. Add a CoordinateSystemSetter
When adding XML data, a coordinate system needs to be set, so to do this, we will use the CoordinateSystemSetter transformer. Add a CoordinateSystemSetter to the canvas and in the parameters, set the Coordinate System to EPSG:4326 and then click OK. 

6. Buffer Points
Now that we have the coordinate system set, we can add a buffered area surrounding the points. This buffer will show us a larger affected area so that the user can avoid the congested traffic that might occur from the incident. 
Add a Bufferer transformer to the canvas and connect it to the CoordinateSystemSetter. In the parameters, set the Buffer Unit Distance to Kilometers and then set the Buffer Unit to 1. Click OK. 


7.  Add KMLPropertySetter
We want to set the properties for the KML file we will be writing out shortly. These properties will appear in the Google Earth navigation tree and when you click on the point the KML balloon pops up. 
Add a KMLPropertySetter transformer to the canvas and connect it to the Output port on the GeographicBufferer. In the parameters, under Navigation Tree, set Name to the category attribute and then set Summary to the description attribute. 
Then, under Description Ballon, set the Content to the description attribute. Click OK to accept the parameters. 


8. Add KMLStyler
The KML now needs to be styled. Add a KMLStyler transformer to the workspace and connect it to the KMLPropertySetter. In the parameters, change the Color and Fill Color to green, then set the Opacity and Fill Opacity to 1. This will make our buffered areas solid green.


9. Clean Up Attributes
One final step before we can write out the data is to remove any excess attributes that we do not need. Add an AttributeRemover transformer to the canvas and open the parameters. In the parameters, click on the ellipsis next to Attributes to Remove and select any of the attributes that start with xml_. 


10. Add a KML Writer
Click on the Writer button to add a new writer to the canvas. For Format, select Google KML, and then for Dataset, browse to a location to save the file and name it incidents.kml. Set the Feature Type Definition to Automatic and then click OK. 


In the Feature Type dialog, set the Feature Type Name to Incidents and click OK. Connect the Incidents writer feature type to the AttributeRemover.
 
11. Run the Workspace
Run the workspace. Browse to the location where you saved the incidents.kml file and double-click on it to open it in Google Earth. 


In Google Earth, notice the name and description in the navigation tree, also click on one of the points to open the information balloon and see the description we added to the balloon. 

Congratulations! You have now read real-time XML data from the web, parsed it into features, and generated georeferenced geometries. You could now write this out to any format you choose simply by adding the appropriate writer (we've used KML in this example) and duplicating the source feature type to the destination.

Data Attribution

The data used here originates from data made available by the British Columbia Government. It contains information licensed under the Open Government License - British Columbia.