Transformers for Working with List Attributes

Introduction

Several FME transformers are specifically designed to work with FME list attributes. In the previous article, we discussed list creation. We will build on that by exploring common transformers used to manipulate, transform, and analyze lists. Approximately 15 such transformers are included in a standard installation, with more options available on the FME Hub.

Other useful transformers include the AttributeManager, AttributeRemover, and AttributeKeeper to clean up and remove redundant lists. It's good practice to clean up lists after finishing with them, resulting in better workspace performance. Remove any excess lists as early in the workspace as possible so that FME only translates essential attributes.

Step-by-Step Instructions

The attached workspace template, ListTransformerExamples.fmwt builds on the scenario from the previous article, where we used list attributes to store all trees within each park. The following examples refer to this workspace when demonstrating how to use list transformers to work with the _trees{} list. Select Run > Run with Feature Caching to run the workspace in a way that allows you to inspect the results of each transformer as features pass through the workspace.

In each section below, we will discuss the transformer’s functionality and walk through the workspace example, illustrating how to use the transformer.

Additionally, if you need a reminder of what List Attributes are, see the documentation.

ListElementCounter

The simplest of the list handling transformers, ListElementCounter returns the number of indexed items within a list. For example, we can use it to count the items within the _trees{} list and store that number in a new attribute, _treeCount.

Since the number of items in the list represents the number of tree points within a particular park, the result tells us how many trees are in each park.
 

ListExploder

The ListExploder is among the top 30 most frequently used FME transformers because it can turn lists into distinct features with corresponding attributes. It explodes a list by creating a separate feature for each indexed list attribute. Most formats cannot write list structures (XML and JSON being notable exceptions), so exploding lists can be useful when elements need to be preserved.

The ListExploder parameters allow you to specify the list that is to be turned into distinct features, indicated by the List Attribute parameter. Optionally, an "Element Index" attribute can be added to each feature to store the position of the element in the original list.



The “Attribute Accumulation” section is where you choose how to merge or preserve the incoming attributes. "Accumulation Mode" has three options:

• Merge List Attributes - retain all attributes and add the exploded list attributes.
• Prefix List Attributes - retain all attributes and add a prefix to the exploded list attributes. This mode is a good choice for retaining all attributes and avoiding attribute name conflicts.
• Only Use List Attributes - preserve the flattened list attributes and remove all original attributes.

"Conflict Resolution" helps resolve problems when an exploded list attribute ends up with the same name as one of the original attributes. You can choose to either preserve the original attribute or give the exploded attribute precedence. If you use Prefix List Attributes, there is no need for conflict resolution.

Example: Exploding List Elements into Distinct Features

When we explode the _trees{} list into distinct features, notice how the features outputted by the ListExploder keep their respective park attributes, as well as the attributes of that particular tree. The ListExploder also adds a new _element_index attribute to indicate the position of this element in the original list.:

Run the workspace and inspect the output to see how the trees are represented in the new table. Note that a few features go into the transformer and many more come out, which indicates that the list has successfully been exploded into distinct features.

Run the workspace and inspect the output to see how the trees are represented in the new table. Notice that a few features go into the transformer and many more come out, which indicates that the list has successfully been exploded into distinct features.

For example, try searching for "English Bay Beach Park" in the table view for the ListExploder input (the TestFilter output) and the ListExploder output.

You can easily toggle between the two tables by holding Ctrl (Command ⌘ for macOS) and clicking on both Visual Preview icons, ensuring you filter for English Bay Beach for both tables.

ListSorter

Sorting lists, whether alphabetically or numerically, makes it easy to pick off certain characteristics from a list. For example, we can use the ListSorter to sort the list by the DIAMETER value to find the largest trees.

Inspect the ListSorter output by viewing the data information window for a park (i.e. English Bay Beach Park). You’ll notice that the window shows the list items are ordered by their diameter, with the tree with the largest diameter at list index 0. 

ListIndexer

The ListIndexer is more selective about turning lists into distinct features. Instead of exploding the entire list like the ListExploder, it only explodes the element(s) at a given index, e.g. _tree{1}: 

This transformer is often placed after another transformer that arranges the list in a particular order before choosing the index, for example, the ListSorter or ListSearcher.

Example: Sorting a List and Selecting the First Element

The ListIndexer can be used to pick the largest tree in the _trees{} list. First, the ListSorter is used to sort the list by the value of  _trees{}.DIAMETER list attribute within each indexed list attribute.

 Next, the ListIndexer is used to get the tree at index 0, which is now the index of the largest tree in the _trees{} list. The result is stored in a set of new attributes with the prefix _largestTree_.

Inspect the ListIndexer in Visual Preview to see the newly created attributes with the _largestTree_ prefix.

ListHistogrammer

The ListHistogrammer builds a histogram from the values in a list and returns these in a new list. The new list is sorted so the value with the most occurrences will be first.

Example: Getting the Feature with the Most Occurrences of a Value

We can use the ListHistogrammer to find the park with the most trees, as well as to find out how many of each species are in each park.

Setting the parameters, Source List Attribute and Histogram List Name, the transformer returns a new list, _speciesHistogram{}. This new list includes count and value data for each indexed list item,  _speciesHistogram{0}, _speciesHistogram{1}, etc.

Inspect the example output by filtering for a park (i.e Grimmett Park) and viewing its attributes in the Feature Information pane. Notice that Grimmett Park contains a _speciesHistogram{} list with count and value data for each of the 15 indexed list items. Moreover, the APPLE SERVICEBERRY tree contains the largest count, and it is sorted to the first indexed list attribute, _speciesHistogram{0}.

ListConcatenator

The ListConcatenator strings the elements of a list together into a single attribute value. In the parameters, you can specify which character to use as a separator (e.g. a comma or a newline character) and the name of the resulting attribute.

Example: Concatenating a List into One Comma-Separated String

The ListConcatenator can be used to create a comma-separated list of tree species in each park. The new attribute is called _species. The example in the workspace refers to a list generated by the ListHistogrammer.



When we inspect the result in the Feature Information pane, we can see the new _species attribute with the concatenated list:

ListSearcher

The ListSearcher allows you to use regular expressions or conditionals to search for patterns in a list and identify which element the pattern was found in.

Example: Searching a List

The Copy Found List Element parameter replicates the functionality of the ListIndexer by copying the list elements into attributes prefixed with _oak_. Notice that the new attributes produced by the ListSearcher are in the Feature Information pane:

Each park contains a _tree{} list, and for each park, only the first match is found. Filtering for “oak” in the example, we see that while there are 125 parks in our dataset, only 37 parks contain an  "Oak" tree.

Remember that filtering in Visual Preview does not alter data in the workspace.

If you want to extract the elements (i.e. TREE_ID, GENUS_NAME, etc.) of interest from the list, you can use a ListIndexer and pass it _list_index as the element to extract, as shown in the example workspace in the ListIndexer_2 transformer.
 

ListSummer

The ListSummer computes the sum of all elements in a list and stores the result in an attribute.

Example: Calculating Averages for List Elements

In the example workspace, we find the average tree height for each park by using ListSummer to determine the total height of all the trees in the _trees{} list, saving the result in the attribute _totalTreeHeight. 

 Then, determine the number of trees in the list using ListElementCounter, storing that output in the attribute _element_count.

Finally, to get the average tree height for each park, divide the two attribute values with the AttributeCreator (you may also notice an easter egg here for a fun height comparison 🦒).

ListRangeExtractor

The ListRangeExtractor returns the minimum and maximum of a list element and saves each as a new attribute.

For example, we can use this transformer to get the smallest and largest tree diameters in each park. 

ListRenamer

The ListRenamer renames or promotes list names or components. Promoting a list component means it will become a simple list instead of a subcomponent of a complex list

For example, _trees{}.SPECIES_NAME is a component (a list attribute)  that can be promoted to the new list, SPECIES_NAME{}. This makes the tree species a list in itself, instead of a subcomponent of the _trees{} list.

ListBuilder

The ListBuilder is a join transformer that merges features based on common attributes. The attributes selected in the Group By parameter become the join key, meaning a feature is output for each combination of values of the specified Group By attributes.

The ListBuilder removes the geometry and only preserves attributes listed in the Group By parameter, so if geometry is required, you can recreate it using a VertexCreator or similar transformer.

Example: Grouping Features Based on Attribute Values

We have tree species and park names and want to find which parks have a specific Genus of tree. We can use the ListBuilder to group features by GENUS_NAME 

In the above parameters, GENUS_NAME is an attribute, not the list attribute, _trees{}.GENUS_NAME

We can also set the "Add To List" parameter to Selected Attributes and choose which attributes to include in the resulting list, _genusList{}. In this case, we chose to preserve COMMON_NAME, GENUS_NAME, SPECIES_NAME, and  park_name:

 When we run the workspace, the original 125 features are merged into 37 features, grouped by their GENUS_NAME.

When we inspect the resulting list in the Feature Information pane, we can see that features have been grouped by Genus. For example, this list includes three different parks that all contain trees of the "ACER" Genus:

A very good alternative to ListBuilder is the Aggregator transformer. Aggregator offers a wider range of functionality, particularly around geometry handling.

ListDuplicateRemover

The ListDuplicateRemover cleans up a list by removing elements with duplicates of a particular attribute value.

For example, in the ListBuilder example above, we can find which parks have a specific Genus of tree while avoiding listing the same parks multiple times. If several oak trees exist in the same park, then by default, the park will be named in the resulting list for every oak tree discovered there, which is redundant.

If we remove duplicate park names, the result looks like this, after the ListConcatenator.

Additional Exercise:  By playing around with these lists and expanding on the example workspace, can you find the rarest Genus and the most widespread Genus?

PythonCaller

Python can be useful for performing more advanced list manipulation. There is a close correlation between Python lists and FME lists – consider the following list of tree names and compare the Python list to the FME list attribute:

_trees    = ["Fernleaf Beech", "Aristocrat Pear", "European Beech", "Pin Oak"]
_trees{0} = Fernleaf Beech
_trees{1} = Aristocrat Pear
_trees{2} = European Beech
_trees{3} = Pin Oak

For example, say we want to search a list for elements that meet two conditions:

1.  It is a Maple tree 
2. It has a canopy diameter greater than 25. 

We could chain together two ListSearchers to check each condition:

This indicates which parks have Maple trees with canopies greater than 25, but it doesn't output a list of all of the qualifying Maple trees. We can achieve this more advanced scenario with Python using the PythonCaller.

Two calls help pass lists in and out of Python. Start with feature.getAttribute():

_treesName = feature.getAttribute('_trees{}.CommonName')

Then perform the desired list manipulation, and then finish with feature.setAttribute():
 

feature.setAttribute('_trees{}.CommonName', _treesList)

The results of PythonCaller for Memorial West Park are in the _treesMaple{} list attributes, COMMON_NAME and DIAMETER for each indexed list element:

You need to expose the Python result attributes in the PythonCaller if you want to work with them in subsequent transformers using the Attributes to Expose parameter.

See Python and FME Basics for a tutorial on working with Python in FME.

List Element Selection Dialog

Many transformers, like the Tester, TestFilter, and AttributeManager, allow you to select an attribute. If you select a list attribute, FME will prompt you with the List Element Selection dialog, where you can choose the element number:

These parameters specify the COMMON_NAME attribute from the element at _trees{0}. If you don't want to hard-code the element number, you can also specify an attribute instead of a number using the Text Editor. For example:

@Value(_treesMaple{@Value(_maple_elements)}.COMMON_NAME)

See the AttributeManager in the Python example above.

Question: Consider the output of the AttributeManager when the above statement is added to

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.