Creating a Unique Identifier | CRCCalculator

Introduction

The CRCCalculator creates a unique ID for a feature based on its geometry and/or attributes. CRC stands for Cyclic Redundancy Check. This is particularly useful if the dataset has a "natural key." Natural keys are created from a list of attribute names and their values for a particular data record or feature. These can be cumbersome to work within an FME workflow (i.e. FeatureMerger) so combining the natural key into a single CRC unique ID can simplify the workspace.

Another use case is to use the CRC value for change detection - to make sure it hasn't been changed, corrupted, deleted, etc. It can also be used as a form of ChangeDetection (instead of using the ChangeDetector to compare geometry, use the Matcher to compare CRC values).

The value could also be used as a guaranteed unique ID number within a dataset.

Example

The attached CRCCalculator_2023.fmw workspace shows an example use of the CRCCalculator transformer.

The CRCCalculator is a great tool for carrying out change detection.

In this example, address records are checked against an updated dataset using CRC values as identification whether a record has been changed. This workspace generates CRC values on-the-fly; an alternative is to save CRC values for each record, to save having to generate them for each run of the workspace. This is the big advantage of using the CRCCalculator over the ChangeDetector.

Other Unique Identifiers to Consider

CRC values are but one type of unique identifier, and while useful, are not guaranteed to be one-of-a-kind from a global standpoint. This is where other unique identifiers may come in handy.

As part of the FME 2023.0 release, the UniqueIdentifierGenerator debuts as the new and improved UUIDGenerator. Previously, the UUIDGenerator could only generate UUIDs or GUIDs. Now, a third option is available with the UniqueIdentifierGenerator: ULIDs. 

What makes ULIDs special? In addition to being able to do everything a UUID can, it also contains a time component which makes it possible to sort by generation time. Compared to UUIDs and GUIDs, ULIDs are more compact which makes it a more attractive choice when working with databases or in situations where storage and efficiency need to be prioritized. 



In most cases the UniqueIdentifierGenerator should suffice for all of your unique identifier needs, however the GUIDGenerator is also worth a mention. This custom transformer is another option if you're working with GUIDs. It can be found in the FME Hub and comes in handy if you're looking to generate Esri-specific GUIDs, which differ from regular GUIDs in that they need to be sandwiched between two curly brackets. This transformer does just that and writes it out as a GFID attribute. 

Something to keep in mind: FME does not use a cryptographically secure method to generate random numbers and as such, these unique identifiers should not be used in situations where data security is crucial.