Introduction
This tutorial describes how to use the SchemaMapper transformer. This transformer remodels the existing feature schema into a desired structure, based on assignment mappings defined in an external lookup table. This technique is very useful when mappings are complex or repetitive, or when the schema needs to be maintained by someone unfamiliar with FME.
Definition of Schema
A schema is the blueprint of a dataset, specifying how data is structured and stored. Each dataset has its own schema, a representation of these specifications. Schema definitions contain feature type names, attribute names, attribute data types, and permitted geometry types. Attribute values, including specialized forms such as domains or enumerated lists, can be configured.
In FME Workbench, the schema can be viewed within the reader’s or writer's properties:
As well, the schema can be viewed on the Workbench canvas by clicking the expand button to expose the list of attributes:
What is Schema Mapping?
The source schema is “what we have”, and the destination schema is “what we want”. Often, the source schema does not match the target schema.
The act of connecting the source schema to the target schema to fulfill the “what we want” is called schema mapping.
In some cases, the source and target schemas correspond, for example, DIAMETER_MM to Diameter. In other cases, values may have to be derived.
General Schema Mapping
Schema mapping can be carried out using common transformers within a workspace. The primary transformer for simple schema mapping is the AttributeManager. To rename attributes to match the target schema, use an AttributeManager transformer to perform the schema mapping.
In the AttributeManager example above, a condition determines the value of the Type attribute. Related schema-mapping transformers include AttributeCreator, AttributeCopier, AttributeRenamer, and AttributeValueMapper.
The values of attributes represented by domains or enumerated lists might also have to be mapped; for example, the Material AttributeValueMapper can be used to map domains:
This process works well for simple transformations, but the SchemaMapper transformer can be used when mappings are potentially complex or repetitive, or when they need to be maintained by someone unfamiliar with FME.
Mapping Schema using SchemaMapper
This transformer converts the existing schema to a new structure, based on mappings defined in an external lookup table. Using a table to define these mappings simplifies completing these tasks.
The schema mapping lookup table, used by the SchemaMapper transformer, defines a series of conditions (filters) and actions to be executed when those conditions are met. The lookup table may come in different formats, such as a comma-separated values (CSV) file, a plain text file, a spreadsheet (Excel or Sheets), or a table in a database such as Oracle, PostgreSQL, Informix, SQL Server, etc.
The SchemaMapper can:
Map attributes: One or more attributes on the feature are mapped by renaming their existing attribute name(s) to a new one. For example:
PipeDiameter → Pipe_Diameter
Map feature types: Feature types, as defined by fme_feature_type, are mapped from their existing values to the new feature type. For example:
WaterMains → Water_Network
Set new attributes: A new attribute is created whose name and value are defined in a lookup table.
Benefits of the SchemaMapper
The following describes the main benefits of using the SchemaMapper:
Powerful: Contains the ability to map attributes, attribute values, and feature types, all from within a single file external to the FME Workspace.
Flexible: Non-FME users can update the process by editing the lookup table directly outside of Workbench.
Intelligent: Filtering capability provides conditional attribute mapping. (for example, if PipeStatus = Active then featureID becomes ActiveID, if PipeStatus = Abandoned then featureID becomes AbandonedID)
Efficient: Multiple source feature types can be routed to the same SchemaMapper for simultaneous mapping.
Extendable: New data can be handled by editing the lookup table rather than the workspace.
User-Friendly: This method allows most mapping to be handled by a single transformer. The non-SchemaMapper method may involve a combination of Tester and/or AttributeValueMapper transformers to handle all individual conditions and lookups.
Tutorial Series
Configuring the SchemaMapper Transformer
The SchemaMapper requires an external lookup table to be configured. This article describes how to create the lookup table and connect it to the SchemaMapper.
SchemaMapper: Attribute Mapping
Attribute mapping is when one or more attributes on a feature are mapped by renaming their existing attribute names to a new name. This article demonstrates how to design a simple lookup table and use it to map attributes.
SchemaMapper: Feature Type Mapping
Feature type mapping is the process of mapping and renaming one or more feature types to a new feature type. This article demonstrates how to perform feature type mapping using a lookup table.
SchemaMapper: Conditional Attribute Mapping
Conditional mapping uses the SchemaMapper’s conditional filtering capabilities. It is a way to define a simple clause to perform attribute mappings based on specific conditions. This article describes how to map a schema based on attribute values within the feature type.
SchemaMapper: Conditional Feature Type Mapping
This article describes how to use the SchemaMapper’s conditional filtering capabilities to define a clause to map feature types based on the value of attributes within the feature type.
SchemaMapper: Advanced Schema Mapping
This article demonstrates some advanced uses of SchemaMapper, including using multiple clauses to filter features, adding new attributes, and modifying existing attribute values. The article also discusses the order of entries in the lookup table, which is useful for debugging.
SchemaMapper: Generating a Lookup Table from Source Data
This article discusses the method for generating lookup tables using data from input feature types.
Additional Resources
- SchemaMapper Documentation
- This example provides thirteen different ways to use the SchemaMapper transformer: SchemaMapper Use Case Example