Files
Introduction
A duplicate feature is one whose geometry exactly matches another feature in the workflow. It usually appears when the same record is submitted to a database twice or when you merge two overlapping datasets.
Several transformers can find duplicates, and the right one depends on what makes two features "the same" in your data, and on how much memory you can spare.
- Matcher: compares geometry, attributes, or both. The only one of these that can use geometry as the matching key.
- CRCCalculator: reduces each feature to a short checksum string, turning a geometry comparison into a string comparison. If the CRC is stored with the data, it never needs to be recalculated.
- DuplicateFilter: splits features into Unique and Duplicate based on one or more key attributes. It cannot key on geometry, but it is feature-based rather than group-based, so it does not hold features in memory. Its documentation describes it as dramatically more efficient than the Matcher for key-based matching.
- ChangeDetector: matches geometry and attributes like the Matcher, but takes two input ports, Original and Revised. It answers a different question: what changed between two datasets. For more info, see our article on Change Detection.
The Matcher is group-based and holds every feature until the last one arrives, because each feature has to be compared against all the others. On a dataset large enough to strain memory, that constraint decides the approach, not the speed of the comparison itself. Part 3 avoids it.
This article explores three approaches on the same dataset, all using the same definition of a duplicate. Part 1 matches on geometry directly. Part 2 replaces the geometry comparison with a checksum. Part 3 combines the checksum with a DuplicateFilter so that no features are held in memory.
Written and captured against FME Form 2026.2
Source Data
The source data is a Precisely MapInfo TAB file of parks in the City of Vancouver, attached to this article (parkspossibleduplicates.zip). It holds 84 features covering Stanley Park, the West End, downtown, and the neighbourhoods around False Creek.
Visually, nothing in this view looks incorrect, but we know there is a duplicate feature. A duplicate feature sits directly on top of the feature it duplicates, so it is hidden. Manually counting polygons on screen will never expose one.
A map view is still worth opening first, because it catches other kinds of problem such as features in the wrong place, missing geometry, or a coordinate system that has not been interpreted correctly.
The table view shows the duplication. Four pairs of features have identical attribute values, including the same ParkId, which makes them candidates for removal.
The repeated ParkId values are 11, 31, 39 and 66.
Four pairs share attributes, but only three of them are duplicates. The two features holding ParkId 11 have different geometry; one is a three-part polygon, the other has two parts. Identical attributes are a reason to analyze further, not an absolute answer. This is why the workflow below matches on geometry rather than on the identifier.
The source data and template workspace can be downloaded from the Files section.
Step-by-Step Instructions
Part 1: Matching Duplicate Geometry with the Matcher
A Matcher transformer is the only option that can treat the geometry itself as the key, so it is helpful start there when you don't trust the attributes to identify a duplicate.
1. Add the Source Data
Start FME Workbench and open an empty canvas. Select Build > Readers > Add Reader and set:
- Format: Precisely MapInfo TAB (MAPINFO/MITAB)
-
Dataset:
ParksPossibleDuplicates.tab- Browse to the downloaded source data file parkspossibleduplicates.zip
Select the .tab file for the Dataset parameter. The reader requires the .dat, .id and .map files alongside it in the same folder, so extract the whole archive rather than pulling out the one file.
2. Add a Matcher
Add a Matcher and connect it to the ParksPossibleDuplicates reader feature type. Set:
-
Check Attributes: Enabled
- Attribute Matching Strategy: Match Selected Attributes
-
Selected Attributes:
ParkId
-
Check Geometry: Enabled
- Match Geometry: 2D
-
Match ID:
_match_id -
Match Count Attribute on SingleMatched:
_match_count
Now, two features with only match when the geometry and the ParkId both agree. This is the definition used in all three parts of this tutorial, so the counts are comparable across them.
Choosing Match Selected Attributes without then populating Selected Attributes leaves the transformer half configured. Check the field is selected before running.
3. Add a StatisticsCalculator
The _match_count attribute answers "how big is each duplicate group". For a single total across the dataset, add a StatisticsCalculator to the Matcher Matched output port and set:
-
Statistics to Calculate:
-
Attribute:
_match_id - Total Count: Enabled
-
Attribute:
Total Count counts the features arriving at the port, so the attribute chosen does not change the number, only the name it is reported under. _match_id is used here because it makes the resulting _match_id.total_count read as a count of matched records.
4. Run Workspace and Inspect Results
Run the workspace with Data Caching enabled, then inspect each Matcher output port.
Data Caching can be enabled under the Run button on the toolbar. With Data Caching enabled, clicking any output port opens that port's features in Data Preview, so you don't need to add an Inspector to view output data . It is be a good idea to disable caching before timing a run or saving a template, since caching adds warnings to the log.
View the following Matcher output ports:
- NotMatched: features with no counterpart anywhere in the dataset.
-
SingleMatched: one copy of each duplicate group, with
_match_countrecording how many features went into it. - Matched: every member of every duplicate group, including the copy you intend to keep.
The Matched output is not a list of records to delete. Both copies of a duplicated pair exist there, including the one you want to keep. The copy we want to keep is the SingleMatched output. The following step writes NotMatched and SingleMatched rather than simply dropping Matched.
Of the 84 input features, 6 Matched, 3 SingleMatched and 78 NotMatched, three duplicate pairs were found. The deduplicated output is the 78 NotMatched plus the 3 SingleMatched, giving 81 features.
The two features holding ParkId 11 are among the 78 in NotMatched. They share every attribute, but their geometry differs, so the Matcher does not call them a match. If the workflow had been keyed on ParkId alone, both would have been reported as duplicates, and one would have been deleted.
On the StatisticsCalculator Summary output port, _match_id.total_count is 6. This is the number of features involved in duplication, not the number of records to delete. The number to delete is 6 minus the 3 groups, so 3 records go (one from each pair).
5. Discard the Duplicates and Write Output
To keep one copy of each duplicate, we will write the Matcher NotMatched and SingleMatched output ports. Select Build > Writers > Add Writer, choose the format you want, and connect both ports to its feature type.
Rather than leaving Matched unconnected, route it to a Logger. The removed features are then written to the log, which gives you something to check if a stakeholder asks what happened to a record. On real data, this is usually worth the extra transformer.
Rerun the workspace to write the data to the output location.
Part 2: Comparing CRC Values Instead of Geometry
Geometry comparison is the expensive part of Part 1. The CRCCalculator reduces each feature to a hexadecimal checksum, then the Matcher compares two short strings instead of two coordinate arrays.
1. Add the Source Data
Open a New workspace, select Build > Readers > Add Reader, and set the following parameters:
- Format: Precisely MapInfo TAB (MAPINFO/MITAB)
-
Dataset:
ParksPossibleDuplicates.tab
2. Add a CRCCalculator
Add a CRCCalculator transformer and connect its Input port to the ParksPossibleDuplicates reader feature type. In the parameters dialog, set the following:
- CRC Algorithm: CRC-32
- Calculate CRC On: Coordinates and Selected Attributes
-
Selected Attributes:
ParkId -
CRC:
_crc
This is the same definition of a duplicate used in Part 1. The geometry and the ParkId. Setting Calculate CRC On to Coordinates and All Attributes instead would fold every other attribute into the key and give different counts.
When the CRC covers coordinates and attributes together, the result is a single string in the form xxxxxxxxyyyyyyyy. The first half is the checksum of the coordinates alone, so with CRC-32 you can match on geometry only by comparing the first eight characters, without recalculating anything. The second half covers the coordinates and attributes together, not the attributes alone, so there is no matching shortcut in the other direction. The MD5 option does not follow this pattern.
3. Add a Matcher
Add a Matcher transformer and connect its Input port to the CRCCalculator Output port. Set the following parameters:
-
Check Attributes: Enabled
- Attribute Matching Strategy: Match Selected Attributes
-
Selected Attributes:
_crc
-
Check Geometry: Enabled
-
Match ID:
_match_id
-
Match ID:
4. Run Workspace and Inspect
Run the workspace with caching enabled and inspect the same three ports as in Part 1 (Matched, SingleMatched, and NonMatched). Both output use geometry and ParkId as a key, so the counts match the previous method; 6 on Matched, 3 on SingleMatched, and 78 on NotMatched.
The ParkId 11 pair stays on NotMatched here as well. The CRC covers the coordinates, so two features whose geometry differs produce different checksums regardless of how identical their attributes are.
5. Discard the duplicates
As in Part 1, add a Writer and connect the Matcher NotMatched and SingleMatched output ports to the writer feature type, leaving the Matched output port unconnected.
Part 3: Filtering Duplicates Without Holding Features in Memory
The Matcher compares every feature against every other feature, so it holds the entire dataset until the last feature arrives. On a large dataset, this is the limiting factor. Once the CRCCalculator has given every feature a comparable key, a DuplicateFilter can separate features one by one instead. It still tracks the key values it has already seen, so memory use is not zero, but it does not hold the features themselves.
1. Build on the Part 2 workspace
Start from the workspace you built in Part 2 and delete the Matcher, leaving the reader and the CRCCalculator connected.
2. Add a DuplicateFilter
Add a DuplicateFilter and connect it to the CRCCalculator Output port. Set:
-
Key Attributes:
_crc - Input is Ordered: No
Leave Input is Ordered set to No unless duplicates are guaranteed to be adjacent in the source. Setting this parameters to Yes compares each feature only against the one immediately before it, so a duplicate anywhere else in the file is passed through as unique, with no warning. It happens to make no difference on this sample because the three duplicate pairs are adjacent.
3. Run Workspace
Run the workspace with caching enabled. The first feature carrying a given CRC exits the DuplicateFilter Unique output port. Each later feature with that same CRC exits the Duplicate port. Unique is the unduplicated dataset, so you can connect a writer to only this port.
81 features pass through the Unique port and 3 pass through Duplicate. Unique is NotMatched plus SingleMatched from Parts 1 and 2, and Duplicate is Matched minus SingleMatched. The DuplicateFilter reports the 3 records to remove, whereas the Matcher reports all 6 records involved.
Choosing a Method
| Approach | Keys on | Holds features in memory | Use when |
|---|---|---|---|
| Matcher alone | Geometry, attributes, or both | Yes, all of them | Geometry is the only reliable key, or you need tolerance and coordinate system checks |
| CRCCalculator and Matcher | Checksum | Yes, all of them | Geometries are large enough that direct comparison is the bottleneck |
| CRCCalculator and DuplicateFilter | Checksum | No | The dataset is too large to hold, and you only need Unique and Duplicate |
The Matcher earns its cost when matching needs to be approximate. Vector Tolerance lets two geometries differ slightly and still match, Lenient Geometry Matching ignores vertex order, and Check Coordinate Systems can require the coordinate system name to agree. A checksum is exact by definition, so none of that is available once you switch to CRC values.
Additional Resources
- Matcher [Documentation]
- CRCCalculator [Documentation]
- DuplicateFilter [Documentation]
- ChangeDetector [Documentation]
- Precisely MapInfo TAB (MAPINFO/MITAB) Reader/Writer [Documentation]
- Data Caching [Documentation]
- Creating a Unique Identifier (CRCCalculator) [Documentation]
- Change Detection [Article] for comparing two datasets rather than deduplicating one
If you run into a problem working through this article, please create a support case.
Data Attribution
The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.