Tutorial: Data Validation and QA with FME

Introduction

While moving, integrating, and transforming data, it's also important to ensure that the data quality is of a suitable standard.

FME has a number of tools and transformers that can be used for data validation. Some - such as the GeometryValidator and AttributeValidator transformers - are specifically designed for data validation. Others - as this tutorial will show - are not specific to data validation, but can be used for that purpose.

When assessing data quality there are three techniques that can apply:

• Identifying Problems: Identifying features that do not conform to the required standard
• Counting Problems: Assessing overall data quality by quantifying the number of substandard features
• Fixing Problems: Improving data quality by fixing the issues found in substandard features

Each of the examples in this tutorial will include information on how to identify, quantify, and fix the problem that is being discussed.
 

Tutorials

Geometric Inconsistencies

These are geometries that are bad in themselves, rather than being part of a substandard network or coverage. This includes:

• Data QA: Identifying Spikes and Outliers with FME
• Data QA: Identifying Self-Intersections with FME

Duplicates

These are features that have some form of duplication. This includes:

• Data QA: Identifying Duplicate Features with FME
• Data QA: Identifying Consecutive Duplicate Vertices with FME and Data QA: Identifying Non-Consecutive Duplicate Vertices with FME
• Data QA: Identifying Duplicate Attribute Values

Small Features

Often very small features are indicative of poor geometry and/or topology. This includes:

• Data QA: Identifying Short Line Features
• Data QA: Identifying Small Polygon Features
• Data QA: Identifying Features Closer than a Minimum Distance (features too close to one another)

Area Topology

These are issues related to a continuous polygon coverage. This includes:

• Data QA Identifying Sliver Overlaps and Gaps in Polygon Coverage

Linear Topology

These are issues relating to a linear network. This includes:

• Data QA: Identifying Bad Topology in Linear Networks

Spatial Concepts

These are geographic features containing some form of logical issue, for example a road represented by a polygon or a bicycle path that runs through a lake. This includes:

• Data QA: Identifying Invalid Spatial Relationships
• Data QA: Identifying Invalid Geometry Types
• Data QA: Invalid Spatial Schemas