SchemaMapper: Advanced Schema Mapping

Introduction

The following is a look at some advanced operations that can be performed utilizing the SchemaMapper transformer. We’ll look at advanced filtering in combination with attribute mapping, attribute setting, and feature type mapping. With great schema mapping comes great responsibility, so we’ll also provide some tips on common errors and debugging complex schema maps.

Process Overview

Multiple Filter Fields

The Filter action sets up a type of ‘IF Statement’ by which FME can match mappings in the lookup table to the features to which the mapping applies. In this example, the SchemaMapper will use multiple filter attributes to define a clause. There are six items defined in the lookup table included in the workspace template:

• MaterialTypeAttribute: The name of an attribute that defines the pipe material.
• MaterialTypeValue: Potential values of pipe material.
• DiameterAttribute: The name of an attribute that defines the pipe diameter.
• DiameterValue: Potential values of pipe diameter.
• StatusAttribute: The name of an attribute that defines the pipe status.
• StatusValue: Potential values of pipe status.

In the lookup table parameters window within the SchemaMapper, the Preview pane shows each conditional clause populated with the attribute values we want to use as filters:

To make use of these, set up multiple filter actions within the SchemaMapper. The filter features action will take care of one pair of attributes and conditional values. The filter features actions should be set up as follows:

Together, these actions create a conditional clause that states:

IF MaterialTypeAttribute = MaterialTypeValue 
AND DiameterAttribute = DiameterValue 
AND StatusAttribute = StatusValue

For example:

IF Material = Steel AND Diameter = 300 AND LifecycleStatus = Active 

Using this set of logical conditions, the SchemaMapper can now take a transformative action based on multiple filtered attributes.

Setting Attribute Values

The SchemaMapper transformer can be used to set attribute values in addition to manipulating the schema type and attributes. Using the “New Attribute” action in the SchemaMapper, attributes and their values can be added to a dataset, or the existing attributes in the schema can be modified.

For example, setting the values for the symbology of a feature:

• autocad_color: The name of an attribute used to define feature color.
• Color: The numeric value equivalent to the desired color
• autocad_lineweight: The name of an attribute in which to define feature lineweight
• Weight: The numeric value equivalent to the desired lineweight.

In order to keep the mapping table simplified, each of the attributes is added as a separate row. There will be more rows in the lookup table, but it will be easier to make edits. You can see this back in the CSV Preview window:

Each Type, Diameter, and Status combination has two rows - one for each symbology attribute. This works because FME doesn’t stop at the first match it finds, but carries on through and executes all mappings that it finds. This technique makes the Mapping Fields dialog simple, as defining one mapping will set multiple attributes.

Add a “New Attribute” action in the SchemaMapper parameters.

When used in conjunction with the filter features action, the dialog will look like this:

Finally, when the workspace is run on Water_Distribution/wMain, this will be the following output:

The features in the output dataset have been assigned different colors/weights based on their pipe attributes (size, status, material). This was done using three Filter actions and a New Attribute action within the SchemaMapper.

Setting Feature Type Mapping in the Table

Feature type mapping can also be included in the Lookup table. Using the format attribute fme_feature_type, this example shows how to add feature type mapping as new rows, similar to how symbology was created above.

For example, feature type mapping has been added to the row:

Material,St,Diameter,450,LifecycleStatus,Active,fme_feature_type,SmallSteelPipes

Which looks like this in the SchemaMapper preview:

The feature types can then be handled with a fanout or similar in the workspace.

Function Calls in Lookup Table

Did you know that FME function calls can be embedded into the value field in a SchemaMapper lookup table? FME will run them when the table is read. For example, a value could be set to:

@UpperCase(@Value(A))

More modern functions need to be wrapped inside the @Evaluate function, like so:

@EvaluateExpression(unused1,<at>DateTimeNow<openparen><closeparen>,unused2)

In FME 2023+, you can also call functions to evaluate value ranges and apply expressions to new mapped values, like so:

FilterAttribute|FilterValue|TargetAttribute|TargetValue

For example:​​​​​​​

AttributeName|fmeexpression=@Evaluate(@Value(AttributeName) >= 25 && @Value(AttributeName) < 75)|NewAttribute|fmeexpression=@Evaluate(@Value(AttributeName) * 10)

Debugging SchemaMapper

The following topics deal with problem-solving when using this transformer. They contain tricks and tips to help solve any issues that may arise.

Mapping Order

The order of lines in the lookup table is important. FME will keep on applying mappings as it works its way from the top down in the lookup table. For example, given these two lines in a lookup table:

Material,Steel,Diameter,450,LifecycleStatus,Active,autocad_color,1 (Red)
Material,Steel,Diameter,450,LifecycleStatus,Active,autocad_color,2 (Yellow)

The feature will first be assigned the color Red and then overwritten with the color Yellow.

This can be useful to know as:

1) There may be a duplicate set of clauses that is causing unexpected mapping.

2) The order can be used to set a “default value”. For example, put this line at the top of the lookup table:

Material,Steel,,,,,autocad_color,1

All features of type “Steel” will match this and be given the color red (regardless of size or status). Then, further in the lookup table, features with a matching Diameter and LifecycleStatus will get set to different values. Therefore, all features in the output that are still coloured red are those that did not match a full filter. These red features can be used to error-check the mapping logic. To put this another way, less specific filters should be placed first in the lookup table, followed by more specific filters.

The key to using this technique is that it will only work when the catch-all is above the other mappings. If it were below, it would reset anything that had already been mapped.

3) Secondary mappings can be set up to simplify a workspace.

For example, take this lookup table:

Material,Steel,NewMaterial,AllPipes
Material,Ductile Iron,NewMaterial,AllPipes
Material,PVC,NewType,AllPipes
Material,AllPipes,autocad_color,5

In this example, certain features are mapped to type AllPipes, and then type AllPipes is mapped to the color blue (#5 in the AutoCAD Color Index). The table maps all the features to a new feature type and then maps that feature type to a new color. This only works if the feature type mapping comes before the color mapping.

Select Row Attribute

The SchemaMapper transformer adds the attribute _schemamapper_row_ids to each mapped feature. This is a list of row numbers in the schema table that were referenced during the mapping.

In some cases, you may have to reduce the size of the schema mapping table to include only those rows that a particular feature type should reference, and then work back from there. The ordering of your rows in the schema table is important - see above. More general filters should be placed first in the lookup table, followed by more specific filters.

You don’t have to use just one SchemaMapper. You could have a lookup table just for domain value mapping and another for feature type and attribute mapping, and include two SchemaMapper transformers in your workspace.

Tutorial Series

Configuring the SchemaMapper Transformer

The SchemaMapper requires an external lookup table to be configured. This article describes how to create the lookup table and connect it to the SchemaMapper.

SchemaMapper: Attribute Mapping

Attribute mapping is when one or more attributes on a feature are mapped by renaming their existing attribute names to a new one. This article demonstrates the design of a simple lookup table and how it can be used to map attributes.

SchemaMapper: Feature Type Mapping

Feature type mapping is when one or more feature types are mapped and renamed to a new feature type. This article demonstrates how to perform feature type mapping using a lookup table.

SchemaMapper: Conditional Attribute Mapping

Conditional mapping uses the SchemaMapper’s conditional filtering capabilities. It is a way to define a simple clause to perform attribute mappings based on specific conditions. This article describes how to map a schema based on the value of attributes within the feature type.

SchemaMapper: Conditional Feature Type Mapping

This article describes how to use the SchemaMapper’s conditional filtering capabilities to define a clause to map feature types based on the value of attributes within the feature type.

SchemaMapper: Generating a Lookup Table from Source Data

The method of generating lookup tables using data sourced from input feature types is discussed in this article.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.