Data QA: Invalid Spatial Schemas

Introduction

A dataset schema (data model) consists of multiple parts. Some parts relate to attributes; others relate to spatial data.

The spatial part of a schema usually defines the feature types (layers, tables, etc.) that exist or are permitted in a dataset, and the geometry types (lines, points, polygons, etc.) that exist or are permitted.

An invalid schema occurs when a feature exists outside the permitted feature types (for example, a layer has a different name than the dataset specifications) or uses a geometry type other than the permitted one (for example, a line feature exists on a layer for polygon features).

This can matter for internal (corporate) consistency and integrity, and when using formats that strictly define table names and permitted geometry types.

FME can handle format limitations automatically, but the user must define whether the data meets a corporate data standard. Because you can run various tests, FME offers transformers to run them. The following example and notes cover just a few of these.

Source Data

The source dataset for this example provides information on construction activity and projects that may affect traffic flow in Vancouver. It is stored in GML format:

In theory, all of these features should consist of simple polygon geometries. Each item should be on a layer representing the organization undertaking the construction. Permitted values are:

• shaw
• hydro
• telus
• city
• fortis
• private
• other

However, we can't be sure the correct layers are used or that the geometry is correct, so we'll need to test it.

Step-by-Step Instructions

Part 1: Locating Invalid Feature Types

Follow these steps to learn how to identify source feature types (layers) that exist in a source dataset. 

1. Start FME Workbench and Add the GML Data

Start FME Workbench and click New to open a blank workspace. To add the source dataset, select Build > Readers > Add Reader from the menu bar and enter the following:

• Format: OGC GML (Geography Markup Language)
• Dataset: Construction.gml
  • To select the dataset, click the ellipsis and browse to the extracted invalidschemadataset folder
• Workflow Options: Single Merged Feature Type

Then click OK. Single Merged Feature Type reads every object into a single layer, which lets us build a list of the layers used.

2. Filter to One Feature per Layer with a DuplicateFilter

Add a DuplicateFilter transformer and connect it to the reader feature type. In the parameters, set:

• Key Attributes: fme_feature_type

3. Write the Layer List with a Text File Writer

Go to Build > Add Writer and enter the following:

• Format: Text File
• Dataset: an output location of your choice

Then click OK.

The Text File writer has a single attribute, text_line_data, and whatever that attribute holds becomes one line of the output file. Our features carry the layer name in fme_feature_type instead, so you need to map the two. That takes two connections, made in order:

First, connect the DuplicateFilter to the Writer. Drag from the DuplicateFilter: Unique port to the Text File writer feature type. 

Then, connect the attributes. Drag from fme_feature_type on the DuplicateFilter to text_line_data on the writer feature type.

4. Run the Workspace and Review the Layer List

Run the workspace. Of the 56 source features, the DuplicateFilter passes 11 to the Unique port and 45 to Duplicate, so the output file has 11 lines, one per layer, both valid and invalid:

Open the output text file. We now have a list of all layers that are used in the source dataset, both valid and invalid:

Five of the seven permitted layers are present: hydro, private, shaw, telus, and fortis. Six more are not: current, misc, Future, xyz, tellus, and _0. Some are more obviously wrong than others. Tellus is a typo for Telus, and _0 happens when a layer name starts with a digit, since GML element names can't.

Part 2: Counting and Filtering Invalid Feature Types

We now have a list of feature types and can see that some are invalid. To count or filter them, we need to know which layers are permitted, and preferably to have that list stored in a file. 

Several transformers can match a feature type against a list, such as AttributeFilter, but here we will use DatabaseJoiner so the permitted values stay in an external file that is easy to maintain.

5. Match Against the Permitted Layers with a DatabaseJoiner

Add a DatabaseJoiner transformer and connect it to a second output from the reader feature type. In the parameters, set:

• Format: CSV (Comma Separated Value)
• Dataset: PermittedLayerList.txt

Then open the Parameters submenu and set:

Fields

• Field Names Line: blank (clear the default value)
• Data Start Line: 1

And press OK to close the Parameters submenu. Finally, under Join, set: 

• Table: CSV
• Join On:
  • Feature Attribute: fme_feature_type
  • Table Field: col0

The permitted layer list has no header row, so the field names line is empty and the data starts on line 1. That also means the single column is named col0.

With this setup, any feature that emerges from the Unjoined port has an invalid feature type, and any feature from the Joined port is on a permitted layer.

6. Count the Invalid Features with a StatisticsCalculator

Add a StatisticsCalculator transformer and connect it to the DatabaseJoiner: Unjoined output port. In the parameters, set:

• Statistics to Calculate: 
  • Attribute: fme_feature_type
  • Statistics: Total Count
    • Click under the Total Count column to add it as a Statistic to Calculate

Your workspace should look similar to this:

7. Run the Workspace and Inspect the Results

Run the workspace and inspect the StatisticsCalculator: Complete output port. Of the 56 source features, 29 are on a layer that isn't permitted. Each is output with its layer recorded in fme_feature_type and the count recorded in the fme_feature_type.total_count attribute, visible in both the Data Preview table (under the attribute column) and the Record Information panel. We can also see the same number reflected in the Data Preview table row count.

We now have a count of the features on invalid layers. We can't correct the layer names because we don't know what they were meant to be, but we have cleaned the dataset by filtering those features out. The 27 features that joined are on permitted layers and carry over to Part 3.

Part 3: Locating Invalid Geometry Types

Follow these steps to identify source features with an incorrect geometry type.

8. Add an Esri Shapefile Writer

Go to Build > Writers > Add Writer and enter the following:

• Format: Esri Shapefile
• Dataset: an output folder of your choice
• Shapefile Definition: Copy from Reader

Then click OK, and connect the new writer feature type to the DatabaseJoiner: Joined port.

9. Configure the Writer Feature Type

Open the parameters for the new writer feature type and set:

• Shapefile Name: fme_feature_type
  • Click the drop-down arrow and select Attribute Value > fme_feature_type
• Geometry: shapefile_polygon

Setting the Shapefile Name from an attribute writes each feature back to a file named after the layer it came from. Setting Geometry to shapefile_polygon tells FME that only polygons are acceptable in those files.

Run the workspace and check the translation log. There are 871 warning messages, which sounds alarming. FME writes the full contents of every rejected feature, so one rejection can produce dozens of lines.

The real figure is 20. Of the 27 features that reached the writer, 7 were written, and 20 were rejected. They failed in two different ways.

Seventeen are linework on layers that should hold areas. That covers all of hydro (11), shaw (3), telus (2) and fortis (1):

Shapefile Writer: Feature type 'shaw' received an incompatible geometry type 'polyline', expected 'polygon'. The following feature cannot be written to this file and will be discarded

The other three fail for a different reason. Each holds both polygon and line geometry in a single aggregate feature, and a feature type restricted to polygons cannot take a mixture:

Dropping heterogeneous aggregate feature for the SHAPEFILE Writer, due to feature type allowed geometries restriction

All three are on private. That is the one layer whose geometry otherwise matches the specification, so a layer can pass the name check in Part 2 and still hold features that fail here.

As long as you saved the workspace in step 1, all rejected features are stored in an FFS (FME Feature Store) format dataset as a form of a spatial log:

Additional Resources

• GML (Geography Markup Language) Reader/Writer: format documentation, including the workflow options used in Part 1
• Esri Shapefile Reader/Writer: format documentation, including the permitted geometry types
• CSV Reader/Writer: format documentation for the reader used inside the DatabaseJoiner

If you're interested in learning more about Data Validation & QA, check out our other articles on the topic or start a discussion in the FME Community.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.