Change Detection

Introduction

Change detection is used to identify differences between a set of input features. It can also identify features that have not undergone changes or are unchanged. Change detection is useful when comparing datasets or databases to see what changes have been implemented in a revised edition.
FME provides users with different transformers that can be used to detect changes depending on the data and what changes need to be detected. Such transformers include: ChangeDetector, UpdateDetector, Matcher, and CRCCalculator.

Change Detection Transformers

ChangeDetector 

The ChangeDetector gives you the option of detecting features that have been:

• Updated
• Inserted
• Upserted
• Deleted
• Unchanged

The ChangeDetector combines the functionality of outdated ChangeDetector (2018 and earlier) and UpdateDetector. More information about the outdated ChangeDetector and deprecated UpdateDetector can be found at the bottom of this article. 

If you have a persistent key (detection key attribute) on both the Original & Revised datasets then the ChangeDetector will add the fme_db_operation attribute to the features. See the article Updating Databases: Using Change Detection to Set fme_db_operation . For line matching, ChangeDetector also includes matching using the Frechet distance - see this blog article on the FME2019+ ChangeDetector

In FME 2022.1, we have introduced the ability to perform UPSERT operations. Find a step-by-step guide in using UPSERT operations in these tutorials. 

• Updating a PostgreSQL Database Using UPSERT
• Updating ArcGIS Online Feature Services using UPSERT

Matcher

The Matcher is a more general approach to change detection and can be more convenient to use in some workflows. The Matcher is able to produce Unchanged (SingleMatched) and Changed (NotMatched) outputs. It can perform matches based on attributes, or attributes and geometry in the example: Detecting Matched Features with the Matcher. For geometry intense datasets, using the Matcher alone may be costly: workflows may benefit from using the Matcher with CRC values as discussed below.

CRCCalculator

Using the CRCCalculator with the Matcher can lessen the cost of the Matcher reading in geometry intense datasets on its own. By pairing the two transformers, the CRCCalculator can create unique values based on coordinates and attributes which then the Matcher can use to compare.

The CRCCalculator can also be used with the ChangeDetector and UpdateDetector to test against the unique values. Additionally, the CRCCalculator can be used with the FeatureMerger and Tester to produce Unchanged, Updated, Deleted, and Inserted outputs as shown in Creating a Unique Identifier.

Change Detection within Databases

Change detection within ArcSDE Geodatabases is possible by comparing the different versions of tables to see changes. This is further detailed in Extracting Differences from an ArcSDE Geodatabase.

Updating Databases shows an example of using the fme_db_operation to perform updates on a PostGIS database. Updates and inserts with the fme_db_operation can also be used within a Smallworld Database as in Update Data in a Smallworld Database.

See also Tutorial: Updating Databases for a more thorough description of database updates

Outdated Transformers 

ChangeDetector (2018 & earlier)

The ChangeDetector is able to detect Added (inserted and updated), Deleted, and Unchanged features between the Original and Revised inputs. Features can be compared using only attributes, only geometry, or both attributes and geometry.

The ChangeDetector (2018 and earlier) cannot distinguish between updates and inserts, if this is required the UpdateDetector outlined below may be used. Depending on the number of features/attributes, the ChangeDetector may require significant processing time.

ChangeDetector has the option of generating a detailed list of changes that gives the attribute or geometry that changed and shows the original vs the revised values.

UpdateDetector (2018 & earlier)

The UpdateDetector (available on FME HUB) is used for translations where datasets have a key attribute to compare against. Features entered through the Revised port are compared against the Original features to detect changes. The UpdateDetector can match 2D, 3D, or no geometry.

By comparing the Original and Revised, the UpdateDetector produces Updated, Inserted, Unchanged, and Deleted features. These ports can be used with the fme_db_operation to change the features accordingly, as demonstrated in Updating Databases: Using Change Detection to Set fme_db_operation.