Files
Introduction
The CRCCalculator creates a unique ID for a feature based on its geometry and/or attributes. CRC stands for Cyclic Redundancy Check. This is particularly useful if the dataset has a "natural key." Natural keys are created from a list of attribute names and their values for a particular data record or feature. These can be cumbersome to work with in an FME workflow (i.e., FeatureMerger), so combining the natural key into a single CRC unique ID can simplify the workspace.
Another use case is to use the CRC value for change detection - to make sure it hasn't been changed, corrupted, deleted, etc. It can also be used as a form of change detection (instead of using the ChangeDetector to compare geometry, use the Matcher to compare CRC values).
You can also use the value as a guaranteed unique ID within a dataset.
Example
The attached CRCCalculator_2023.fmw workspace shows an example use of the CRCCalculator transformer.
The CRCCalculator is a great tool for carrying out change detection.
In this example, address records are checked against an updated dataset using CRC values to identify whether a record has been changed. This workspace generates CRC values on-the-fly; an alternative is to save CRC values for each record to avoid having to generate them for each run of the workspace. This is the big advantage of using the CRCCalculator over the ChangeDetector.
Other Unique Identifiers to Consider
CRC values are one type of unique identifier, and while useful, they aren't guaranteed to be one-of-a-kind globally. That is where other unique identifiers can help.
As part of the FME 2023.0 release, the UniqueIdentifierGenerator debuts as the new and improved UUIDGenerator. Previously, the UUIDGenerator could only generate UUIDs or GUIDs. Now, the UniqueIdentifierGenerator adds a third option: ULIDs.
What makes ULIDs special? In addition to doing everything a UUID can, it includes a time component, which makes it possible to sort by generation time. Compared to UUIDs and GUIDs, ULIDs are more compact, making them a more attractive choice for databases or situations where storage and efficiency take priority.
style="color: #000000;" style="border-style: none; display: inline-block; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; height: 163px; text-decoration: none; white-space: pre-wrap; width: 338px;"
In most cases, the UniqueIdentifierGenerator should suffice for your unique identifier needs; however, the GUIDGenerator is also worth mentioning. This custom transformer is another option if you're working with GUIDs. You can find it in the FME Hub, and it comes in handy if you want to generate Esri-specific GUIDs, which differ from regular GUIDs because they need to be sandwiched between two curly brackets. This transformer does just that and writes it out as a GFID attribute.
FME does not use a cryptographically secure method to generate random numbers, and as such, these unique identifiers should not be used in situations where data security is crucial.