How Blocking (or Feature-Holding) Transformers Alter the Flow of Features in FME

Introduction

The flow of features through an FME workspace is not task-driven, but rather feature-driven. Features are read into a workspace one at a time, and are pushed onto the next workspace object as they’re read. There is no waiting for all features to be read before moving onto the next translation task. 

Whereas most FME transformers obediently follow this core flow-of-data rule, blocking transformers unabashedly break these rules in the name of functionality. Take the Sorter transformer as an example of this rebellious behavior. It would be rather silly to try and sort a list of surnames alphabetically without first having the whole list of surnames available for the Sorter to act upon. Sometimes, features need to wait patiently at a transformer’s input port before being allowed to proceed, despite how interruptive this waiting might be to the flow of data.   

Other examples of true FME blocking transformers include:
 

• the FeatureHolder, whose sole purpose is to gather all input features together in one spot without actually performing any further action on the features; and

• the Neighborfinder, whose actions cannot be accurately performed without taking the spatial relationships amongst all input features into consideration 

This article and accompanying template workspace use the Sorter transformer to demonstrate how blocking (or “feature-holding”) transformers cause the flow of data to be temporarily interrupted in a workspace. This interruption allows features which are read into the workspace later than others to catch-up with their cohorts at the input port of these transformers. In this way, the relevant information from all input features can be acquired and used at once by these feature-holding rebels. 
 

Step-By-Step Instructions

This demonstration makes use of the BlockingTransformers_2022.fmwt workspace provided above. Please download and save this template workspace locally to follow along with the steps described below.


1) Open BlockingTransformers_2022.fmwt in FME Workbench 2022.2.2 or newer

Once the workspace is open in FME Workbench, save it locally as a .fmw (File → Save). This action will also create a copy of the data associated with the template workspace to the same location as this newly saved .fmw file.

The workspace itself is a fairly simple one. It was designed to read linear features which represent the heights of a group of people, and then sort these features based on the value of a height attribute.



Note: the Junction transformer was put in place to demonstrate relevant events in the translation only. It is not necessary to include a Junction in order to take advantage of blocking transformers. 


2) Add a Breakpoint to the Workspace

Right-click on the connection between the reader feature type, People, and the Junction transformer. Choose the Add Breakpoint menu option.



Breakpoints allow for viewing how an individual feature moves through a workspace. Here, we will use a breakpoint to observe how the flow of features is interrupted at the Sorter transformer.


3) Run the Workspace with Breakpoints Enabled

Access the Run menu from the Workbench ribbon, and enable the Stop At Breakpoints option. Check and make sure that Enable Feature Caching is disabled; breakpoints and feature caching are not compatible with each other. Also ensure Enable Feature Counting is enabled.



Now save and run the workspace. Notice that a single feature is read from the source data, and then a pause in the translation occurs at the breakpoint. A Feature Inspector dialog then opens to allow inspection of this first feature. 



If the Feature Inspector dialog is docked to the main FME Workbench window, it can be undocked by simply clicking on its tab (at the bottom of its dialog) and dragging it until it becomes its own stand-alone window. This may make it easier to view the movement of each feature through the workspace.



Undocking the Feature Inspector window 


4) Step to the Next Connection from the Feature Inspector Window. 

At the bottom of the Feature Inspector window are several buttons: three with green triangle icons, and one with a red square icon. These buttons control how a user can follow a feature through the workspace. Hover over each icon to see a tooltip describing its action.

Now select the Step to next connection icon. This action moves the feature on to the next connection in the workspace. After selecting this icon, notice how the connection between the reader feature type and the Junction in the demo workspace is no longer highlighted in red, while the connection between the Junction and the LengthCalculator transformer is highlighted. The red highlighting denotes that the feature being followed has now arrived at the input port of the LengthCalculator. 



Click Step to next connection again, and notice how the highlighted connection is now between the LengthCalculator and the Sorter transformers. This denotes that the feature has emerged from the output port of the LengthCalculator and has traveled to the input port of the Sorter. Also note that this feature now has a Height attribute, visible in the Contents pane of the Feature Inspector.




5) Return the Translation Flow to the Reader Feature Type 

Recall that the Sorter is a blocking transformer, meaning that it will pause features at its input port, holding them in memory until all source features have arrived. Now that the first feature has arrived at the input port of the Sorter, it is reasonable to expect the translation flow to return to the reader feature type so that the next feature can be read into the translation.

Click Step to next connection, and observe where the highlighted connection occurs in the demo workspace.



As expected, the flow of this translation returns to the reader, indicated by the red highlighting now shifting to the connection between the reader and the Junction. The next feature has been read into the workspace. This behavior of returning the flow to the reader once a feature arrives at the input port of a blocking transformer is the key distinguishing feature of these transformers. This behavior supports the action(s) that these transformers perform, where the correctness of their output(s) depends on receiving information from every source feature first. 


6) Select Continue to next breakpoint from the Feature Inspector dialog

Click the Continue to next breakpoint icon (to the right of Step to next connection). Since there is only one breakpoint in this workspace, this action will result in the current read-in feature moving along the translation until it reaches the input port of the Sorter. Then the flow will return first to the reader, where the next feature will enter the translation, and then stop at the breakpoint.
 
Now, to view what happens once the final source features are read into the workspace, an adjustment to the Breakpoint settings is needed. Click the red square Stop translation icon in the Feature Inspector dialog. Next, right-click on the breakpoint between the reader feature type and the Junction. Select Edit Breakpoint from the pop-up menu. 



In the Edit Breakpoint dialog that opens, change the Stop parameter from Always to When hit count is, then choose equal to from the adjacent drop-down options, and then enter 108 in the final text input box. This change will allow the translation to proceed until the number of features read into the workspace is 107, and the 108th feature is waiting to be read.



Now re-run the translation, still with Stop at Breakpoints enabled, and the translation will pause once the feature count number displayed between the reader and the Junction reaches 107. Notice how the order in which features are read into the workspace is identical to the order in which features appear in the source dataset. 


7) Examine the Translation Log

The source dataset for this demo workspace contains 110 features. At this point, 107 features have been fully processed up to the Sorter transformer. The final few features will be advanced through the workflow manually, to show the changes that occur in the Translation Log text as the reader finishes its processing. 

Navigate to the Translation Log window (or activate it from View → Windows → Translation Log) and be sure you can view both this window and the Feature Inspector window clearly.  Scroll to the bottom of the Translation Log text, and keep an eye on this part of the log as you step through the final few features of this translation.

In the Feature Inspector window, click Step to next connection three times to advance the next feature through to the Sorter. Click Step to next connection three more times, watching the Translation Log text as you do so. Notice the additional message which populates in the log text each time another feature is processed by the reader. 

Now, to advance the final feature through the reader and on to the Sorter, click Step to next connection another three times. Be sure to pay close attention to the text at the bottom of the Translation Log window as you do this.   
As the final feature emerges from the Sorter, notice how the Translation Log window updates with a few more lines of information. The most important message is the line Emptying factory pipeline, which denotes that the reader has finished reading all source data and has now terminated its actions.



Also notice the information from the LengthCalculator has now been logged, followed by the messages that the Sorter has started and completed sorting all 110 features.


8) Send Features to the Inspector Transformer

Select Step to next connection once more. Now notice how the connection line between the Sorter and the Inspector is highlighted in red and shows a feature count of one. In the Translation Log, there will now be a message indicating the initiation of this Inspector has occurred. This denotes that the first sorted feature has arrived at the Inspector, and is being stored in memory. 



Note: this Inspector transformer can be replaced with any writer feature type, and the translation log text at this point would show the initiation of the writer instead of the Inspector.


9) Select Continue translation in the Feature Inspector 

Finally, click on the Continue Translation button in the Feature Inspector window to run the remainder of the translation without stopping at any more breakpoints. Inspect the final output in the Table view of the Visual Preview window. Notice how features are now ordered based on their Height value, in descending order as specified by the sort condition outlined in the Sorter.



One thing to keep in mind about blocking transformers is the fact that they hold features in system memory as features reach their input port. This means that, for very large datasets, there will be a large demand on system memory with the potential of running low on or out of memory during translations involving blocking transformers.

If these large datasets can be broken down into smaller pieces, with extraneous attributes removed and / or complex geometries simplified, memory issues may be avoided when running a translation which involves blocking transformers.