SchemaMapper: Advanced Schema Mapping

Introduction

The following is a look at some advanced operations that can be performed utilizing the SchemaMapper transformer. We’ll look at advanced filtering in combination with attribute mapping, attribute setting, and feature type mapping. With great schema mapping comes great responsibility, so we’ll also provide some tips on common errors and debugging complex schema maps.

Step-by-step Instructions

Multiple Filter Fields

The Filter action sets up a type of ‘IF Statement’ by which FME can match mappings in the lookup table to the features to which the mapping applies. In this example, the SchemaMapper will use multiple filter attributes to define a clause. There are six items defined in the lookup table included in the workspace template:

• MaterialTypeAttribute: The name of an attribute that defines the pipe material.
• MaterialTypeValue: Potential values of pipe material.
• DiameterAttribute: The name of an attribute that defines the pipe diameter.
• DiameterValue: Potential values of pipe diameter.
• StatusAttribute: The name of an attribute that defines the pipe status.
• StatusValue: Potential values of pipe status.

In the lookup table parameters window within the SchemaMapper, the Preview pane shows each conditional clause populated with the attribute values we want to use as filters:

Step 1 - To make use of these, set up multiple Filter Actions, within the SchemaMapper. Each Filter Features Action will take care of one pair of attributes and conditional values. The Filter Features Actions should be set up as below:

Together, these actions create a conditional clause that states:

IF MaterialTypeAttribute = MaterialTypeValue 
AND DiameterAttribute = DiameterValue 
AND StatusAttribute = StatusValue

For example:

IF Material = Steel AND Diameter = 300 AND LifecycleStatus = Active 

Using this set of logical conditions, the SchemaMapper can now take a transformative action based on multiple filtered attributes.

Setting Attribute Values

In addition to manipulating the schema type and attributes, the SchemaMapper transformer can be used to set attribute values. Using the “New Attribute” action in the SchemaMapper, attributes and their values can be added to a dataset or the existing attributes in the schema can be modified.

For example, setting the values for symbology of a feature:

• autocad_color: The name of an attribute in which to define feature color.
• Color: The numeric value equivalent to the desired color
• autocad_lineweight: The name of an attribute in which to define feature lineweight
• Weight: The numeric value equivalent to the desired lineweight.

In order to keep the mapping table simplified, each of the attributes is added as a separate row. There will be more rows in the lookup table, but it will be easier to make edits. You can see this back in the CSV Preview window:

Each combination of Type, Diameter and Status has two rows - one for each symbology attribute. This works because FME doesn’t stop at the first match it finds, but carries on through and executes all mappings that it finds. This technique makes the Mapping Fields dialog simple, as defining one mapping will set multiple attributes.

Step 2 - Add a “New Attribute” action in the SchemaMapper parameters.

When used in conjunction with the filter features action, the dialog will look like this:

Finally, when the workspace is run on Water_Distribution/wMain, this will be the following output:

The features in the output dataset have been assigned different colors/weights based on their pipe attributes (size, status, material). All of this was done using three Filter actions and a New Attribute action within the SchemaMapper.

Setting Feature Type Mapping in the Table

Feature Type mapping can also be included into the Lookup table. Using the format attribute fme_feature_type, this example shows how to add Feature Type mapping as new rows, similar to how symbology was created above.

For example, Feature Type mapping has been added to the row:

Material,St,Diameter,450,LifecycleStatus,Active,fme_feature_type,SmallSteelPipes

Which looks like this in the SchemaMapper preview:

The feature types can then be handled with a fanout or similar in the workspace.

Function Calls in Lookup Table

Did you know that FME function calls can be embedded into the value field in a SchemaMapper lookup table? FME will run them when the table is read. For example, a value could be set to:

@UpperCase(@Value(A))

More modern functions need to be wrapped inside the @Evaluate function, like so:

@EvaluateExpression(unused1,<at>DateTimeNow<openparen><closeparen>,unused2)

In FME 2023, you can also call functions to evaluate value ranges and apply expressions to new mapped values, like so:

FilterAttribute|FilterValue|TargetAttribute|TargetValue

For example:​​​​​​​

AttributeName|fmeexpression=@Evaluate(@Value(AttributeName) >= 25 && @Value(AttributeName) < 75)|NewAttribute|fmeexpression=@Evaluate(@Value(AttributeName) * 10)

Debugging SchemaMapper

The following topics deal with how to problem solve when using this transformer. It contains tricks and tips to help solve any issues that may arise.
 

Mapping Order

The order of lines in the lookup table is important. FME will keep on applying mappings as it works its way from the top down in the lookup table. For example, given these two lines in a lookup table:

Material,Steel,Diameter,450,LifecycleStatus,Active,autocad_color,1 (Red)
Material,Steel,Diameter,450,LifecycleStatus,Active,autocad_color,2 (Yellow)

The feature will first be assigned the color Red, and then it will be overwritten by the color Yellow.

This can be useful to know as:

1) There may be a duplicate set of clauses which is causing unexpected mapping.

2) The order can be used to set a “default value”. For example, put this line at the top of the lookup table:

Material,Steel,,,,,autocad_color,1

All features of type “Steel” will match this and be given the color red (regardless of size or status). Then further in the lookup table, features with a matching Diameter and LifecycleStatus will get set to different values. Therefore, all features in the output which are still coloured red are those which did not match a full filter. These red features can be used as a way to error check the mapping logic. To put this another way, less specific filters should be placed first in the lookup table, followed by filters which are more specific.

The key to using this technique is that it will only work when the catch-all is above the other mappings. If it were below then it would reset anything which had already been mapped.

3) Secondary mappings can be setup to simplify a workspace.

For example, take this lookup table:

Material,Steel,NewMaterial,AllPipes
Material,Ductile Iron,NewMaterial,AllPipes
Material,PVC,NewType,AllPipes
Material,AllPipes,autocad_color,5

In this example, certain features are mapped to type AllPipes, and then type AllPipes is mapped to the color blue (#5 in the AutoCAD Color Index). The table maps all the features to a new Feature Type and then maps that Feature Type to a new color. This only works provided the Feature Type mapping comes before the color mapping.

Select Row Attribute

The SchemaMapper transformer adds the attribute _schemamapper_row_ids to each mapped feature. This is a list of row numbers in the schema table that were referenced during the mapping.

In some cases you may have to reduce the size of the schema mapping table to include only those rows that a particular feature type should reference and then work back from there. The ordering of your rows in the schema table is important - see above. More general filters should be placed first in the lookup table with more specific filters following.

You don’t have to use just one SchemaMapper. You could have a lookup table just for domain value mapping and another for feature type and attribute mapping and include two SchemaMapper transformers in your workspace.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.