Clearing Blocking Transformers

Introduction

This example is intended to show what parameters can clear a blocked transformer.

Step-by-step Instructions

1) Start FME Workbench and open the workspace (or template). Notice that we are reading a dataset of parking meters and calculating the minimum, maximum, and average parking rate for each neighborhood; this is done using a Group-By set to Neighborhood.

Right-click on the connection between the StatisticsCalculator and the Logger/Inspector transformer. Choose the option to Add Breakpoint:

2) Select Run > Run with breakpoints from the menubar:

Be sure that Feature Caching is turned OFF! Breakpoints and feature caching are not compatible with each other.

Now run the workspace. Notice that all source features are read from the source dataset so that 8602 features are read (and stored in memory) before the first output feature is generated.

Click the “Continue to Next Breakpoint” button to cause each feature to be output one at a time. Do this for a handful of features and then click the Stop button to stop the translation.

NB: If the feature count stops at 8144 originally, it’s because the display has not yet been updated to show the actual number of features read. The log message “Emptying factory pipeline” confirms that the data has now all been read, the display is just to catch up.

3) So the StatisticsCalculator is a blocker transformer. It saves up all of the meter features until it has them all, so that it can then form groups. The only way we can “unblock” the transformer and save resources is if the data was already pre-sorted into neighborhood order.

So, open the source data in the Data Inspector and check the Neighborhood column in the Table View. It should show that the data is already sorted by neighborhood in the following order: Downtown, Fairview, Kitsilano, Mount Pleasant, Strathcona, West End.

4) Back in Workbench set the StatisticsCalculator parameter Input Ordered to By Group:

Re-run the translation. This time notice that the transformer starts to output features before all of the data has been read, so it is having to store less data in memory. Each click on “Continue to Next Breakpoint” causes a new feature to be output and the next group of features to be read.

If you wish, add a Sampler transformer after the reader, to let only every 10th feature (or less) pass by. That way you'll have to click the next button fewer times.

Lesson: By using the Input Ordered parameter, the StatisticsCalculator processes one group of features at a time, before the next group has even been read. However, it does require that the source data already exists in the correct order.

NB: If you check the peak memory used, will it be less? In theory, yes. In practice no, but that’s partly because we’re only working with small amounts of data and partly because we’re using the StatisticsCalculator:Summary port - which allows us to use breakpoints but skews the memory usage for some reason.

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.