Tutorial: SchemaMapper Transformer

Introduction

This tutorial describes using the SchemaMapper transformer. This transformer converts the existing schema of features to a new structure, based on mappings defined in an external lookup table. This technique is very useful when the mappings are complex, repetitive or when they need to be maintained by someone who is not familiar with FME.
 

Definition of Schema

Schema is the structure of the dataset. Each dataset has its own schema which is a physical representation of the data. Schema primarily defines the feature type name, attribute names, attribute data types, and permitted geometry type. Also, attribute values in the form of domains or enumerated lists may have tp be handled.

In FME Workbench the Schema can be viewed within the Reader’s or Writer's properties:



As well, the schema can be viewed on the Workbench canvas by clicking the expand button to expose the list of attributes:

What is Schema Mapping?

The source schema is “what we have” and the destination schema is “what we want”.  Usually, the source schema  ("what we have"), does not match the target schema ("what we want").

The act of connecting the source schema to the target schema, to fulfill the “what we want”, is called Schema Mapping.

In some cases there is a correspondence between the source and target schemas, for example DIAMETER_MM to Diameter. In some cases values may have to be derived.
 

General Schema Mapping

Schema Mapping can be carried out using common transformers within a workspace. The primary transformer for simple schema mapping is the AttributeManager.  To rename attributes too match the target schema use an AttributeManager transformer to perform the Schema Mapping.


In the AttributeManager example above, a conditional is used to derive the Type attribute value. Other related schema mapping transformers include the AttributeCreator, AttributeCopier, AttributeRenamer and AttributeValueMapper transformers. 
The values of attributes that are represented by domains or enumerated lists might also have to be mapped, for example the Material. AttributeValueMapper can be used to map domains:

This process works well for simple transformations but when mappings are potentially complex, repetitive or when they need to be maintained by someone who is not familiar with FME, then the SchemaMapper transformer can be used.

Mapping Schema using SchemaMapper

This transformer converts the existing schema of features to a new structure, based on mappings defined in an external lookup table. Using a table to define these mappings simplifies the completion of these tasks.

The schema mapping lookup table, used by the SchemaMapper transformer, defines a series of conditions that are to be met (filters), and a series of actions that will be executed when the conditions are met. The lookup table may come in different formats such as: a comma-separated or plain text file; a spreadsheet (Excel or Google); or as a table in a database such as Oracle, PostGRES, Informix, SQL Server, etc.

There are three main actions that can be taken by the SchemaMapper:

Map Attributes:
One or more attributes on the feature are mapped by renaming their existing attribute name (s) to a new one. For example:

PipeDiameter ? Pipe_Diameter

Map Feature Types:
Feature types – as defined by fme_feature_type – are mapped from their existing value, to one which defines the new feature type. For example:

WaterMains ? Water_Network

Set New Attributes:
A new attribute is created whose name and value are defined in a lookup table.
 

Benefits of the SchemaMapper

The following describes the main benefits of using the SchemaMapper:

Powerful: Contains the ability to map attributes, attribute values, and Feature Types, all from within a single file external to the FME Workspace.

Flexible: Non-FME users can update the process by editing the lookup table directly outside of Workbench.

Intelligent: Filtering capability provides conditional attribute mapping. (for example, if PipeStatus = Active then featureID becomes ActiveID, if PipeStatus = Abandoned then featureID becomes AbandonedID)

Efficient: Multiple source Feature Types can be routed into the same SchemaMapper to provide simultaneous mapping functionality.

Extendable: New data can be handled by editing the lookup table – and not the workspace.

User-Friendly: Allows most mapping to be handled by a single transformer. The non-SchemaMapper method may involve a combination of many Tester and/or AttributeValueMapper transformers to handle all of the individual conditions and lookups.
 

Tutorial Series

Configuring the SchemaMapper Transformer

The SchemaMapper requires an external lookup table in order to be configured. This article contains a description of creating the lookup table as well as connecting it to the SchemaMapper.
 

SchemaMapper: Attribute Mapping

Attribute mapping is when one or more attributes on a feature are mapped by renaming their existing attribute name(s) to a new one. This article demonstrates the design of a simple lookup table and how it can be used to map attributes.
 

SchemaMapper: Feature Type Mapping

Feature Types Mapping is when one or more Feature Types are mapped to a new Feature Type and renamed. This article demonstrates how to perform Feature Type mapping using a lookup table.
 

SchemaMapper: Conditional Attribute Mapping

Conditional mapping makes use of the SchemaMapper’s conditional filtering capabilities. It is a way to define a simple clause to perform attribute mappings based on specific conditions. This article describes how to map schema based on the value of attributes within the Feature Type.
 

SchemaMapper: Conditional Feature Type Mapping

This article describes how to use the SchemaMapper’s conditional filtering capabilities to define a clause in order to map Feature Types based on the value of attributes within the Feature Type.
 

SchemaMapper: Advanced Schema Mapping

An article which demonstrates some of the advanced uses of the SchemaMapper, including the use of multiple clauses to filter features, how to add new attributes and how to modify existing attribute values. The article also discusses the order of entries in the lookup table, which is useful for debugging.
 

SchemaMapper: Generating a Lookup Table from Source Data

How to generate lookup tables using data sourced from input feature types is discussed in this article.
 

Additional Resources

• SchemaMapper Documentation
• This example provides thirteen different ways to use the SchemaMapper transformer: SchemaMapper Use Case Example