Data QA: Identifying Consecutive Duplicate Vertices

Introduction

A duplicate vertex (duplicate point) occurs when a geometry has one or more vertices that occur multiple times within the feature. Duplicate vertices are those with identical X, Y, and Z coordinates, to the same number of decimal places as in the data.

Duplicate vertices are not only a sign of lower-quality data but can also be a data format problem. Some formats permit duplicate vertices (for example, MicroStation DGN allows zero-length lines), while others prohibit them (for example, Oracle Spatial).

The duplicate vertex might occur sequentially in the geometry (for example, A, B, C, C, D, E) or out of sequence (A, B, C, D, C, E). It might be duplicated once (A, B, C, C, D), or duplicated multiple times (A, B, C, C, C, D, C, E, C).

Sometimes a duplicate vertex is valid. For example, the start and end points of a polygon should be identical if it is to close properly (A, B, C, D, E, A). Sometimes a linear feature loops around and rejoins at a midpoint (A, B, C, D, E, C). So it is not always easy to identify invalid features on this basis alone.

There are various FME transformers that can be used to identify duplicate vertices, but some transformers — or combinations of transformers — are much more efficient than others.

• GeometryValidator: identifies and fixes duplicate vertices that occur consecutively within a single geometry.
• ClosedCurveFilter: identifies features that form a closed loop, and can therefore be used to detect (or eliminate from suspicion) features with duplicate end points.
• CoordinateExtractor: extracts a list of coordinates from a feature, which can then be analyzed to look for duplicates.

In general, GeometryValidator is used more often because consecutive duplicate vertices are a more obvious issue.

However, CoordinateExtractor is better at detecting out-of-sequence duplicate vertices, enabling further investigation.

This example uses the GeometryValidator transformer to identify duplicate points in sequence. A second example uses a combination of ClosedCurveFilter and CoordinateExtractor to identify nonconsecutive duplicate points.

In this scenario, the data will be validated, issues will be recorded, and the data will be cleaned before it is put into production use.

The source data is a MicroStation Design file, buildings.dgn, containing line features that represent building footprints.

Step-by-Step Instructions

Part 1: Locating Consecutive Duplicate Vertices

Follow these steps to learn how to locate consecutive duplicate vertices with a GeometryValidator transformer.

1. Add Source Data

Open FME Workbench and begin with an empty canvas. On the top menu, click Add Reader. In the Add Reader dialog, set the following:

• Format: Bentley MicroStation Design (V8)
• Dataset: buildings.dgn
• Parameters:
  • Advanced:
    • Remove Duplicate Points: Disabled

Ensure this parameter is turned off. As part of Data QA, we want to identify where and how many duplicate vertices there are.

Click OK to add the reader. If prompted, select the BuildingFootprints level as the data to read.

2. Inspect the Data

Click the BuildingFootprints reader feature type on the canvas to open the Mini Toolbar. Click View Source Data to view the data in Visual Preview. The data looks correct at a glance, and it is difficult to identify any duplicate vertices.

3. Add a GeometryValidator Transformer

Add a GeometryValidator transformer to the canvas and connect the BuildingFootprints MicroStation V8 reader to it. Open the transformer parameters and set the following:

• Issue to Detect: Duplicate Consecutive Points
• Attempt Repair: No

Duplicate points can be fixed, but for now, the workflow only identifies their locations.

4. Inspecting the GeometryValidator Output

Click the GeometryValidator to open the Mini Toolbar, then click Run To This. Once the workspace finishes running, click the green inspect button next to each output port to inspect the output in Visual Preview. Two features exit through the InvalidParts port. These features have been flagged as invalid. The duplicate points are highlighted through the IssueLocations port.

Part 2: Counting Consecutive Duplicate Vertices

Counting the number of invalid features and bad points is easy because they have already been filtered out. For example, FME Workbench feature counts show the numbers involved:

The StatisticsCalculator transformer can be used to generate data QA insights as an attribute, such as counts of duplicate values.

Follow these steps to learn how to count duplicate vertex features.

1. Add a StatisticsCalculator

Add a StatisticsCalculator transformer to the workspace and connect it to the GeometryValidator Failed port.

Open the transformer parameters and set the following:

• Statistics to Calculate:
  • Attribute: Any Attribute (e.g. _issues{0}.count)
  • Total Count: Enabled

2. Re-run the Workspace

Run the workspace again. This time, the output includes an attribute indicating how many bad features there are for each type. To change the name of the output attribute from StatisticsCalculator, use the AttributeRenamer transformer.

Part 3: Fixing Consecutive Duplicate Vertices

Fixing invalid duplicate vertices is straightforward: delete one of the duplicates. The GeometryValidator has an option to do this.

1. Repairing the Duplicate Vertices

Open the GeometryValidator parameters dialog and set the following:

• Attempt Repair: Yes

2. Re-run the Workspace

Run the workspace again. This time, no vertices fail, but the duplicate points still output separately as Repaired, showing where they existed.

Additional Resources

Data QA: Identifying Non-Consecutive Duplicate Vertices [Article]

Data QA: Identifying Duplicate Features [Article]

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.