Tutorial: SchemaMapper Transformer

Introduction

This tutorial describes using the SchemaMapper transformer. This transformer remodels the existing schema of features to a desired structure, based on assignment mappings defined in an external lookup table. This technique is very useful when the mappings are complex or repetitive, or when the schema needs to be maintained by someone who is not familiar with FME.

Definition of Schema

A schema is the blueprint of a dataset, containing the specification that defines how data is structured and stored. Each dataset has its own schema, a representation of these specifications. Schema definitions contain feature type names, attribute names, attribute data types, and permitted geometry types. Attribute values, including specialized forms such as domains or enumerated lists, can be handled through configuration.  

In FME Workbench, the schema can be viewed within the reader’s or writer's properties:


As well, the schema can be viewed on the Workbench canvas by clicking the expand button to expose the list of attributes:

What is Schema Mapping?

The source schema is “what we have”, and the destination schema is “what we want”.  Often, the source schema does not match the target schema.

The act of connecting the source schema to the target schema, to fulfill the “what we want”, is called schema mapping.

In some cases, the source and target schemas correspond, for example, DIAMETER_MM to Diameter. In other cases, values may have to be derived.

General Schema Mapping

Schema mapping can be carried out using common transformers within a workspace. The primary transformer for simple schema mapping is the AttributeManager.  To rename attributes to match the target schema, use an AttributeManager transformer to perform the schema mapping.


In the AttributeManager example above, a condition derives the Type attribute value. Related schema mapping transformers include the AttributeCreator, AttributeCopier, AttributeRenamer, and AttributeValueMapper transformers. 

The values of attributes represented by domains or enumerated lists might also have to be mapped, for example, the Material. AttributeValueMapper can be used to map domains:

This process works well for simple transformations, but the SchemaMapper transformer can be used when mappings are potentially complex or repetitive, or when they need to be maintained by someone unfamiliar with FME.

Mapping Schema using SchemaMapper

This transformer converts the existing schema to a new structure, based on mappings defined in an external lookup table. Using a table to define these mappings simplifies the completion of these tasks.

The schema mapping lookup table, used by the SchemaMapper transformer, defines a series of conditions that are to be met (filters) and a series of actions that will be executed when the conditions are met. The lookup table may come in different formats, such as a comma-separated or plain text file, a spreadsheet (Excel or Sheets), or a table in a database such as Oracle, PostgreSQL, Informix, SQL Server, etc.

The SchemaMapper can:

Map attributes: One or more attributes on the feature are mapped by renaming their existing attribute name(s) to a new one. For example:

PipeDiameter → Pipe_Diameter

Map feature types: Feature types, as defined by fme_feature_type, are mapped from their existing value to one that defines the new feature type. For example:

WaterMains → Water_Network

Set new attributes: A new attribute is created whose name and value are defined in a lookup table.

Benefits of the SchemaMapper

The following describes the main benefits of using the SchemaMapper:

Powerful: Contains the ability to map attributes, attribute values, and feature types, all from within a single file external to the FME Workspace.

Flexible: Non-FME users can update the process by editing the lookup table directly outside of Workbench.

Intelligent: Filtering capability provides conditional attribute mapping. (for example, if PipeStatus = Active then featureID becomes ActiveID, if PipeStatus = Abandoned then featureID becomes AbandonedID)

Efficient: Multiple source feature types can be routed into the same SchemaMapper to provide simultaneous mapping functionality.

Extendable: New data can be handled by editing the lookup table, not the workspace.

User-Friendly: This method allows most mapping to be handled by a single transformer. The non-SchemaMapper method may involve a combination of many Tester and/or AttributeValueMapper transformers to handle all of the individual conditions and lookups.

Tutorial Series

Configuring the SchemaMapper Transformer

The SchemaMapper requires an external lookup table to be configured. This article describes how to create the lookup table and connect it to the SchemaMapper.

SchemaMapper: Attribute Mapping

Attribute mapping is when one or more attributes on a feature are mapped by renaming their existing attribute names to a new one. This article demonstrates the design of a simple lookup table and how it can be used to map attributes.

SchemaMapper: Feature Type Mapping

Feature type mapping is when one or more feature types are mapped and renamed to a new feature type. This article demonstrates how to perform feature type mapping using a lookup table.

SchemaMapper: Conditional Attribute Mapping

Conditional mapping uses the SchemaMapper’s conditional filtering capabilities. It is a way to define a simple clause to perform attribute mappings based on specific conditions. This article describes how to map a schema based on the value of attributes within the feature type.

SchemaMapper: Conditional Feature Type Mapping

This article describes how to use the SchemaMapper’s conditional filtering capabilities to define a clause to map feature types based on the value of attributes within the feature type.

SchemaMapper: Advanced Schema Mapping

This article demonstrates some of the advanced uses of the SchemaMapper, including using multiple clauses to filter features, adding new attributes, and modifying existing attribute values. The article also discusses the order of entries in the lookup table, which is useful for debugging.

SchemaMapper: Generating a Lookup Table from Source Data

The method of generating lookup tables using data sourced from input feature types is discussed in this article.

Additional Resources

• SchemaMapper Documentation
• This example provides thirteen different ways to use the SchemaMapper transformer: SchemaMapper Use Case Example