Transformers for Working with List Attributes

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2022.0

Introduction

A number of FME transformers are designed to work with list attributes. In the previous article, we discussed list creation. In this article, we will discuss the top transformers that are used to manipulate, transform, and analyze lists. There are about 15 transformers in total that manipulate list attributes, plus more on FME Hub.

Other useful transformers include the AttributeManagerAttributeRemover, or AttributeKeeper to clean up and remove redundant lists. It's good practice to clean up lists after finishing with them, resulting in better workspace performance. Remove any excess lists as early in the workspace as possible so that FME only translates essential attributes.

Step-by-Step Instructions

The attached workspace template ListTransformerExamples.fmwt builds on the scenario from the previous article, where we used list attributes to store all trees within each park. The below examples refer to this workspace when demonstrating how to use list transformers to work with the _trees{} list. Select Run > Run with Feature Caching to run the workspace in a way that allows you to inspect the results of each transformer as features pass through the workspace.

In each section below, we will discuss what the transformer does and walk through an example of how to use it.
 

ListElementCounter

The simplest of the list handling transformers,  ListElementCounter returns the number of elements in a list. For example, we can use it to count the elements in the _trees{} list and store that number in the _treeCount attribute.
2022-07-13_13-18-08.png

Since the number of elements in the list represents the number of tree points within a particular park, the result tells us how many trees are in each park.
 

ListExploder

The ListExploder is in the top 30 most frequently used FME transformers because of its power to turn lists into distinct features. It explodes a list by creating a separate feature for each list element. Most formats do not have the ability to write list structures (XML and JSON being notable exceptions), so exploding lists can be useful when elements need to be preserved.

Example: Exploding List Elements into Distinct Features

The ListExploder parameters allow you to specify the list that is to be turned into distinct features. Optionally, an "Element Index" attribute can be added to each feature to store the position of the element in the original list.
2022-07-13_13-20-14.png

The “Attribute Accumulation” section is where you choose how to merge or preserve the incoming attributes. "Accumulation Mode" has three options:

  • Merge List Attributes - retain all attributes and add the exploded list attributes.
  • Prefix List Attributes - retain all attributes and add a prefix to the exploded list attributes. This mode is a good choice for retaining all attributes and avoiding attribute name conflicts.
  • Only Use List Attributes - preserve the flattened list attributes and remove all original attributes.

"Conflict Resolution" helps resolve problems when an exploded list attribute ends up with the same name as one of the original attributes. You can choose to either preserve the original attribute or give the exploded attribute precedence. If you use Prefix List Attributes, there is no need for conflict resolution.

Check the ListExploder documentation for more details on how to configure these parameters.

When we explode the _trees{} list into distinct features, note how the features outputted by the ListExploder keep their respective park attributes, as well as the attributes of that particular tree:
2022-07-13_13-27-03.png
The ListExploder also adds a new _element_index attribute to indicate the position of this element in the original list.

Run the workspace and inspect the output to see how the trees are represented in the new table. Note that a few features go into the transformer and many more come out, which indicates that the list has successfully been exploded into distinct features.
 

ListSorter

Sorting lists, whether alphabetically or numerically, makes it easy to pick off certain characteristics from a list. For example, we can use the ListSorter to sort the list by the Diameter attribute to find the largest trees.
2022-07-13_14-16-54.png

 

ListIndexer

The ListIndexer is more selective about turning lists into distinct features. Instead of exploding all list elements like the ListExploder, it only explodes the element(s) at a given index, e.g. {1}. This transformer is often placed after another transformer that arranges the list in a particular order before choosing the index – for example, the ListSorter or ListSearcher.

Example: Sorting a List and Selecting the First Element

The ListIndexer can be used to pick the largest tree in the _trees{} list. First, the ListSorter is used to sort the list by the _trees{}.Diameter attribute. Next, the ListIndexer is used to get the tree at index 0. The result is stored in a set of new attributes with the prefix _largestTree_.
2022-07-13_14-08-12.png

2022-07-13_14-10-32.png
 

ListHistogrammer

The ListHistogrammer builds a histogram from the values in a list and returns these in a new list. The new list is sorted so the value with the most occurrences will be first.

Example: Getting the Feature with the Most Occurrences of a Value

We can use the ListHistogrammer to find the park with the most trees, as well as to find out how many of each species are in each park.
2022-07-13_14-45-34.png

With these parameters set, the transformer will count how many of each tree name exists in the _tree{} list. It returns the _speciesHistogram{} list. Note how the resulting list includes .count and .value list attributes:
2022-07-13_14-47-16.png

Inspect the output and select a park (i.e Grimmett Park) and view its attributes in the Feature Information pane. Grimmett Park contains a _speciesHistogram{} list with two elements. It shows that the park contains 2 APPLE SERVICEBERRY trees and 1 JAPANESE FLOWERING CHERRY tree.
 

ListConcatenator

The ListConcatenator strings the elements of a list together into a single attribute value. In the parameters, you can specify which character to use as a separator (e.g. a comma or a newline character) and the name of the resulting attribute.

Example: Concatenating a List into One Comma-Separated String

The ListConcatenator can be used to create a comma-separated list of tree species in each park. The new attribute is called _species. The example in the workspace refers to a list generated by the ListHistogrammer.
2022-07-13_14-12-48.png

When we inspect the result in the Feature Information pane, we can see the _species attribute with the concatenated list:
2022-07-13_14-14-44.png

 

ListSearcher

The ListSearcher allows you to use regular expressions or conditionals to search for patterns in a list and identify which element the pattern was found in.

Example: Searching a List

We can use this transformer to search the _trees{} list for elements with "Oak" in the COMMON_NAME. The index of the first element that contains the string is saved in an attribute called _list_index.
2022-07-13_14-19-03.png

The Copy Found List Element parameter replicates the functionality of the ListIndexer by copying the list elements into attributes prefixed with _oak_. Note the new attributes produced by the ListSearcher:
2022-07-13_14-20-54.png

Each park contains a _tree{} list, and for each park, only the first match is found. In this scenario, there are 125 parks, and 37 parks contain an  "Oak" tree.

If you want to extract the element of interest from the list, you can use a ListIndexer and pass it _list_index as the element to extract as shown in the example workspace.
 

ListSummer

The ListSummer computes the sum of all elements in a list and stores the result in an attribute.
2022-07-13_14-36-06.png

For example, for each park, we can determine the total height of all the trees, and then divide it by the number of list elements to get the average tree height in that park.
 

ListRangeExtractor

The ListRangeExtractor returns the minimum and maximum of a list element.
2022-07-13_14-37-32.png

For example, we can use this transformer to get the smallest and largest tree diameters in each park.
 

ListRenamer

The ListRenamer renames or promotes list names or components. Promoting a list component means it will become a simple list instead of a subcomponent of a complex list, e.g. _trees{}.Species can be promoted to Species{}. This makes the list of tree species list in itself, instead of a subcomponent of the _trees list.
2022-07-13_15-59-46.png
 

ListBuilder

The ListBuilder is a join transformer that merges features based on common attributes. The attributes selected in the Group By parameter become the join key, meaning a feature is output for each combination of values of the specified Group By attributes.

Note the ListBuilder removes the geometry and only preserves attributes listed in the Group By parameter, so if geometry is required, you can recreate it using a VertexCreator or similar transformer.

Example: Grouping Features Based on Attribute Values

We have tree species and park names and want to find which parks have a specific genus of tree in them. We can use the ListBuilder to group features by GENUS_NAME2022-07-13_16-16-51.png:

We can also set the "Add To List" parameter to Selected Attributes and choose which attributes to include in the resulting list, _genusList{}. When we run the workspace, the original 125 features are merged into 37 features, grouped by their GENUS_NAME:

When we inspect the resulting list in the Feature Information pane, we can see that features have been grouped by Genus. For example, this list includes three different parks that all contain trees of the "ACER" genus:
2022-07-13_16-19-53.png

A very good alternative to ListBuilder is the Aggregator transformer. Aggregator offers a wider range of functionality, particularly around geometry handling.
 

ListDuplicateRemover

The ListDuplicateRemover cleans up a list by removing elements with duplicates of a particular attribute value.
2022-07-13_16-22-27.png

For example, in the ListBuilder example above, we can find which parks have a specific Genus of tree while avoiding listing the same parks multiple times. If several oak trees exist in the same park, then by default the park will be named in the resulting list for every oak tree discovered there, which is redundant.
If we remove duplicate park names, the result looks like this, after the ListConcatenator.
2022-07-13_16-25-56.png

Additional Exercise:  By playing around with these lists, and expanding on the example workspace, can you find the rarest Genus and the most widespread Genus?
 

PythonCaller

Python can be useful for performing more advanced list manipulation. There is a close correlation between Python lists and FME lists – consider the following list of tree names and compare the Python list to the FME list attribute: 

_trees    = ["Fernleaf Beech", "Aristocrat Pear", "European Beech", "Pin Oak"]
_trees{0} = Fernleaf Beech
_trees{1} = Aristocrat Pear
_trees{2} = European Beech
_trees{3} = Pin Oak


For example, say we want to search a list for elements that meet two conditions: it is a Maple tree and it has a canopy diameter greater than 25. We could chain together two ListSearchers to check each condition:

1585268648600.png

This indicates which parks have Maple trees with canopies greater than 25, but it doesn't output a list of all of the qualifying Maple trees. We can achieve this more advanced scenario with Python using the PythonCaller.

Two calls help pass lists in and out of Python. Start with feature.getAttribute():
 

_treesName = feature.getAttribute('_trees{}.CommonName')


Then perform the desired list manipulation, and then finish with feature.setAttribute():
 

feature.setAttribute('_trees{}.CommonName', _treesList)


Download the attached workspace template, ListTransformerExamples.fmwt, to see the complete workspace and Python code (towards the bottom of that workspace). The results of the workspace are in the _treesMaple{} list atrribute:
2022-07-14_14-54-06.png
For the West Memorial Park. 
You need to expose the python result attributes in the PythonCaller if you want to work with them in subsequent transformers using the Attributes to Expose parameter.

See Python and FME Basics for a tutorial on working with Python in FME.
 

List Element Selection Dialog

Many transformers, like the Tester, TestFilter, and AttributeManager, allow you to select an attribute. If you select a list attribute, FME will prompt you with the List Element Selection dialog, where you can choose the element number:

Screen Shot 2021-05-21 at 10.52.44 AM.png
These parameters specify the CommonName attribute from the element at _trees{0}. If you don't want to hard-code the element number, you can also specify an attribute instead of a number using the Text Editor. For example:

@Value(_treesMaple{@Value(_maple_elements)}.COMMON_NAME)

See the AttributeCreator in the Python example above.

 

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.

 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.