Data QA: Identifying Short Line Features

Introduction

Short lines are those whose length is less than a specified tolerance. These can be found through the combination of measuring the length of lines and then applying a test condition with a filter transformer.

Testing for short lines is a good QA test because lines less than a certain length are usually indicative of problems such as duplicate vertices, invalid network edges, dangles (overshoots), and generally poor linework.

For example, denoted by the arrows below are two extremely short lines, dangling from where the road network was clipped against a polygon boundary that didn't quite match:

It's very simple to count how many of these bad features exist. However, as discussed below, fixing lines like this automatically is more difficult.

Source Data

The source dataset for this example is a set of lines (in an AutoCAD DWG dataset) representing roads in the city of Vancouver.

The dataset looks like this in the FME Data Inspector:

The scenario here is that we wish to create a proper edge/node network, but recognize that we should add some QA checks to ensure that no bad linework is being used.

Step-by-Step Instructions

Part 1: Locating Short Lines

Follow these steps to learn how to identify short line features.
 

1. Start FME Workbench and Add Source Data

Start FME Workbench and click New to open a blank workspace. 

To add the source dataset, select Build > Readers > Add Reader from the menu bar.

In the Add Reader dialog, enter the following:

• Format: Autodesk AutoCAD DWG/DXF
• Dataset: Roads.dwg 

Click the Parameters button and set:

• Group Entities By: Attribute Schema

Click OK and OK again to add the reader.
 

2. Calculating Length of the Lines
The source dataset is made up of line features. To test their length we first have to measure them using a LengthCalculator transformer.

So, add a LengthCalculator transformer. Connect all of the reader feature types to the LengthCalculator:Input port.

3. Filtering Features Based on Length
To assess the length of a feature requires a transformer that can filter features based on a particular condition. The easiest to use - and most common - transformer is the Tester.

So, add a Tester transformer and connect it to the LengthCalculator Output port.
 

4. Setup Tester Parameters
How short a line needs to be before it becomes suspect is subjective. However, a road link shorter than it is wide seems like an obvious candidate. Wikipedia tells us that an average road lane is 3 meters wide. Most roads in Vancouver have two lanes. Therefore, we should flag all road segments that are less than six meters in length.

So, open the Tester parameters dialog. Set a test condition to check for road features whose measured length is less than 6:

• Left Value: _length
• Operator: <
• Right Value: 6

5. Inspect the Results
Run the workspace and inspect the output by clicking on the magnifying glass next to each Tester output port. The output will appear in the Data Preview window. Using the Display Control panel, adjust the feature colors and width to make the short lines stand out more. The features can also be viewed and customized in the FME Data Inspector, using the Open in Data Inspector option. 

Examine the red features to see whether they are truly problem features and why. You'll find there are isolated problems spread throughout the city.
 

6. Identifying the Type of Roads Affected
Let's pretend we want to identify the type of roads that are affected; i.e. are they Private, Residential, Secondary, etc. The first step to doing so is to expose that information as an attribute.

Click the gear button on any source feature type to open its properties dialog. Click the Format Attributes tab and locate the attribute fme_feature_type.

• Format Attributes:
  • Name: fme_feature_type
  • Exposed: enabled

This will make the fme_feature_type attribute available within the workspace.
 

7. Inspecting by Type of Road
Add an Inspector to the canvas. Connect it to the Tester Passed port. Open the parameters dialog for the Inspector.

• Group Processing: Enabled
  • Group By: fme_feature_type

Re-run the workspace. Each type of road will be separated into its own layer in the Data Preview window or FME Data Inspector. You'll see that of the 14 bad features, 4 are arterial roads, 7 are residential, and 3 are secondary.

Part 2: Counting Short Lines

Counting the number of bad features is quite easy because we have already filtered them out. For example, even the feature counts show us the numbers involved:

To create a count stored in an attribute is simple using the StatisticsCalculator transformer.

Follow these steps to learn how to count short line features.
 

8. Create a Count Stored in an Attribute 
Add a StatisticsCalculator between the Tester Passed port and the Inspector. Open the parameters dialog and set:

• Calculation Method: Numeric
• Summary Accumulation Mode: Drop Incoming Attributes 
• Statistics to Calculate: Total Count
• Attribute: _length
  • It doesn't really matter which attribute we select, since we only want a count of features.

Click OK to close the dialog.

The output attribute will be called _length.total_count. To rename this attribute, add an AttributeRenamer after the StatisticsCalculator, then open the parameters and set: 

• Input Attribute: _length.total_count
• Output Attribute: BadFeatures

Re-run the workspace. This time the output should include an attribute that denotes how many bad features there are of each type.

If you connected the StatisticsCalculator:Summary output port to the Inspector, there will only be a single output feature. To get all output features, ensure that the Complete port is connected.

9. Group the Count Based on Road Type
If you want a count of the bad features (as an attribute) based on the road type, re-open the StatisticsCalculator parameters dialog and set:

• Group Processing: Enabled
  • Group By: fme_feature_type

Run the workspace, and you will see the number of bad features for each type of road.

Part 3: Fixing Short Lines

There is no simple solution to fix short lines automatically because short lines can occur in many different ways. The simplest case is a short line that should be merged into its neighbor using the LineCombiner transformer, but as the following example shows, automatically fixing short lines can sometimes introduce more errors than it repairs and become quite complex.

12. Add a LineCombiner Transformer
Add a LineCombiner transformer to the workspace and insert it between the reader feature types and the LengthCalculator transformer:

13. Set Up the LineCombiner parameters 
The LineCombiner has a number of parameters to control how it operates. One danger here is unwanted connections, since there is no way to tell it that only the short features should be joined.

Open the LineCombiner transformer's parameters dialog and set: 

• Combine On Attributes: HBlock
• Consider Node Elevation: Yes
  • Found under Advanced Parameters

Now lines will not be connected unless they are part of the same road, and not if they pass over each other at a different elevation.
 

14. Fix Incidentally Joined Lines
Even making use of the LineCombiner parameters, some lines may get inadvertently connected together. For example, where two segments in the same block should be split by an incoming connection but are now joined together:

To avoid this, add an Intersector transformer between the LineCombiner and LengthCalculator transformers.

15. Re-run the Workspace

You will see that some of the problem features have been cleaned up. For example, look for the feature with a StreetId of 11255. Before running the cleanup workspace, it was an unnecessary short section. Afterward, the LineCombiner and Intersector have merged the feature and intersected the block at the junction with StreetId 11342.

However, other features have been unnecessarily intersected (see StreetId 13545):

So this is a case where the intersection solves some problems, but causes others.
 

16. Re-join Erroneously Intersected Features 
Now we need to re-join features that were erroneously intersected, leaving those that were intersected because they were part of an original merge. This is where things start to get more complex, but we can tell one feature from another by comparing the number of parts it was broken into to the number of parts it had after the LineCombiner.

Open the parameters dialog for the LineCombiner:

• Attribute Accumulation:
  • Generate List: Enabled
    • List Name: MyList
    • Add To List: All Attributes

This creates an FME list, an attribute with multiple values, that tells us which (and how many) road segments are being joined.

17. Count Elements

Add ListElementCounter and Tester transformers, and a second LineCombiner - all connected at the end of the workspace like so:

Open the ListElementCounter parameters dialog:

• List Attribute: MyList

That will record the number of parts to each feature originally joined together.

Open the Tester_2 parameters and use the following values in the test clause: 

• Left Value: _element_count
• Operator: =
• Right Value: _segments

This will tell us whether to join features back together. i.e. if the LineCombiner and Intersector had no effect on the number of parts, then it should not have been altered and should be fixed.

Finally, open the LineCombiner_2 parameters dialog and set:

• Combine On Attributes: StreetId

This will join lines based on StreetId, putting back together streets that were unnecessarily connected.
 

18. Run the Workspace and Inspect the Outputs

You may - if you wish - add a third LengthCalculator/Tester combination to see whether there are any remaining short lines:

Notice that the output now has most of the original short lines fixed.

You may also have noticed that there are now more short lines than originally! This is because this technique has split up lines at intersections where there was previously no node. In other words, it's found lines that had been incorrectly noded and fixed those too - revealing some short lines in the process.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.