SchemaMapper: Generating a Lookup Table from Source Data

Introduction

When starting a schema mapping project, it may be helpful to load all the feature type data into lookup tables for use with the SchemaMapper transformer. This creates a great starting point and eliminates the need to copy data manually. This technique should dramatically speed up the schema mapping setup process, especially when multiple input files are mapped.

This article demonstrates two separate methods for building a lookup table. Each lookup table is separated by the type of schema mapping being performed: Attribute Mapping, Feature Type Mapping, and Filtering.

Examples

Attribute and Feature Type Lookup Tables

In this example, the attributes of a Keyhole Markup Language (KML) file will be mapped using the SchemaMapper. The files will then be read into the workspace using the merge filter.

The Schema (Any Format) reader extracts all the attribute names from the files and writes them to the lookup table. Since the reader extracts the attributes into a list, a ListExploder transformer grabs each attribute name.

At the same time, a separate table is generated listing each input Feature Type’s name using the incoming fme_feature_type name attribute.

Example feature type table:

Example attribute table:

The resulting lookup tables both contain a blank column enabling a user to add their own schema mapping rules. Once completed, the lookup table can be processed by SchemaMapper transformer.

Filter Lookup Table

This type of lookup table is used with the SchemaMapper’s Filter Action. In order to set up a filter, the SchemaMapper must know attribute names and their unique values for the Feature Type. This workspaces produces a lookup table that contains unique values for each attribute contained by the input feature type.

In this example, the same KML file is going to be used to generate the filter lookup table.

First, the dataset is read into the workspace using a Generic reader. An AttributeKeeper is used to retain only the necessary data (do not retain unnecessary data such as OBJECTID for shapefiles) and then each attribute is exploded and duplicates are removed, for this example DateInstalled will not be selected. Finally, the data is sorted and written to the lookup table.

AttributeKeeper is used so that unexposed attributes do not end up in the filter lookup table. AttributeRemover will retain unexposed attributes.

Example filter lookup table:

In the lookup table output, extra blank columns (Source, Destination) are added, enabling users to add their schema mapping rules. Once completed, the lookup table can be processed by the SchemaMapper transformer

Tutorial Series

Configuring the SchemaMapper Transformer

The SchemaMapper requires an external lookup table to be configured. This article describes how to create the lookup table and connect it to the SchemaMapper.

SchemaMapper: Attribute Mapping

Attribute mapping is when one or more attributes on a feature are mapped by renaming their existing attribute names to a new one. This article demonstrates the design of a simple lookup table and how it can be used to map attributes.

SchemaMapper: Feature Type Mapping

Feature type mapping is when one or more feature types are mapped and renamed to a new feature type. This article demonstrates how to perform feature type mapping using a lookup table.

SchemaMapper: Conditional Attribute Mapping

Conditional mapping uses the SchemaMapper’s conditional filtering capabilities. It is a way to define a simple clause to perform attribute mappings based on specific conditions. This article describes how to map a schema based on the value of attributes within the feature type.

SchemaMapper: Conditional Feature Type Mapping

This article describes how to use the SchemaMapper’s conditional filtering capabilities to define a clause to map feature types based on the value of attributes within the feature type.

SchemaMapper: Advanced Schema Mapping

This article demonstrates some of the advanced uses of the SchemaMapper, including using multiple clauses to filter features, adding new attributes, and modifying existing attribute values. The article also discusses the order of entries in the lookup table, which is useful for debugging.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.