Data Inspection in FME

Introduction

The flow of features through an FME workspace is not task-driven but rather feature-driven. Features are read into a workspace one at a time and pushed onto the next workspace object as they’re read. There is no waiting for all features to be read before moving on to the next translation task. 

Since the data flows through the workspace as individual features, this makes it easy to inspect the data at any point in the workspace. There are different ways to view data in FME: 

• Initial Data Inspection 
• Feature Caches 
• Breakpoints - Individual Features
• Inspector Transformers 
• Written Output Data 
• Logging

Initial Data Inspection

When working with any data source in FME, it is typically a good idea to view the data so that you know what you are working with. Data can be viewed outside of a workspace using the FME Data Inspector application that comes with FME Form. For more information on viewing data with the FME Data Inspector, see Getting Started with FME Form: Introduction to FME Data Inspector. 

Data can also be viewed directly in FME Workbench after adding a reader feature type to the canvas or with certain transformers, such as the FeatureReader or DatabaseJoiner, that add data in the middle of a translation. 

To view data, single-click on the feature reader type or FeatureReader transformer to open the mini menu, then click on the View Source Data button. The data can then be viewed in Visual Preview within FME Workbench. 

Some formats require the workspace to run first before the data can be accessed. If the View Source Data button is disabled or does not return data, run the workspace first and follow the steps in the Feature Cache section.

Feature Caching 

Feature caching is a method of viewing data output by each object on the canvas, such as a reader or writer feature types, transformers, or junctions. There are several different reasons to use feature caching while authoring a workspace: 

• Initial data inspection
• After changing a parameter in a transformer 
• Only running sections of a workspace
• Caching the response from a database or API
• Troubleshooting
• Confirming the output before writing the data out or submitting an API call 
• Caching data for use in certain transformer parameters, such as the AttributeExposer

Feature caching does temporarily store data at each object, which can affect performance or disk space. Feature caching should be turned off for production workspaces or if experiencing a significant reduction in performance. For testing purposes, a Sampler transformer can be used to reduce the number of features when working with large datasets. 

Enable Feature Caching

To enable feature caching, click the drop-down arrow next to the Run button in FME Workbench and select Enable Feature Caching. Once feature caching is enabled, when the workspace is run, caches will appear on each output port of each object on the canvas if there is data. 

Viewing Feature Caches

If feature caching is enabled, after running the workspace, green icons will appear on each output port that has data. To view the feature cache, click on the green icon, and the data will open in Visual Preview. 

If the feature cache is yellow, that means the cache is stale and has not been updated via running the workspace. A stale cache can occur from a parameter changing upstream of the cache or by a change in which canvas object is connected.

Breakpoints

Breakpoints are inspection points where a single feature can be viewed at a time, whereas feature caching is used to view all output features at a certain output port. Breakpoints are manually inserted wherever feature inspection needs to occur. Breakpoints are most commonly used for troubleshooting but can also be used to inspect a single feature to decide what action needs to be taken next. 

Enabling Breakpoints

Breakpoints are inserted on the connection lines between objects on the canvas. To insert a breakpoint, right-click on a connection line and select Add Breakpoint. 

A red octagon will appear on the connection line to indicate the presence of a breakpoint. A workspace can have as many breakpoints as needed for inspection purposes. 

Running with Breakpoints

Once at least one breakpoint has been added to the canvas, click the drop-down arrow next to the Run button and select Stop at Breakpoints. Once Stop at Breakpoints has been enabled, click the Run button to start running the workspace. 

With Stop at Breakpoints enabled a new Feature Inspector window will appear, this is where each feature can be viewed as it stops at the breakpoint. If the feature has geometry, it can be viewed in the top half of the window, and any attributes can be viewed in the bottom half. The breakpoint controls are at the bottom of the window. The first button will continue the feature to the next object whether or not there is a breakpoint. The second button will continue running the translation to the next breakpoint, and the third will continue the translation ignoring all the breakpoints. For more information on the Feature Inspector window, please see the documentation. 

Breakpoints can also be set up to use test clauses to only have the feature stop should the test be satisfied or only stop after a certain number of features. To enable tests or stop counts, right-click on the breakpoint and select Edit Breakpoint. In the Edit Breakpoint dialog, either enable Test Clauses or modify the Stop Count. 

There are some transformers in FME that wait until all features have arrived before releasing all of the features to the next object. For example, the Sorter transformer, as it is inefficient to sort a list without first obtaining all the items in the list. Breakpoints can be used to see these types of transformers more clearly. Other blocking transformers include the FeatureHolder, NeighborFinder or any transformer with Group Processing enabled.

Inspector Transformer

The Inspector transformer can be connected to any output port and will open data in Visual Preview when a workspace is run. This is similar to running a workspace with feature caching enabled; however, it is much less memory intensive, as only the data that reaches the Inspector will be viewed. 

The Inspector transformer can also be connected to multiple output ports at once to layer data in Visual Preview. For example, it can be used to view the park boundaries where the drinking fountains intersect. 

View Output Data

If a writer feature type is connected to the workspace or a FeatureWriter, after a workspace is run, the output data can be viewed directly in FME via Visual Preview. To view the output data in FME, single-click on the writer feature type or FeatureWriter to open the mini menu, then click on View Written Data. 

Written data can also be accessed via the computer’s file explorer. To quickly find written data, single-click on the writer feature type and click Revel in Finder (macOS) or Open File Explorer (Windows OS). 

Certain tasks in FME result in manipulating web data via an API, in this case, data won’t be “written” out to a format or database. To record these various calls, a JSON writer or Text File Writer could be used, or a Logger transformer could also be used attached to the HTTPCaller or various connector type transformers 

Logging

Sometimes, it isn’t the individual features that need to be inspected but something about the workspace itself. Each time a workspace is run locally, a .log file is created. However, by default, the log is overwritten each time the workspace is run. This can be disabled via Workspace Options. For more information on workspace logging, see the documentation. 

Features can also be logged throughout the workspace by using the Logger transformer and attaching it to any output port. The Logger can enhance error messages or log certain features to aid in troubleshooting. When working with APIs, the Logger can document the calls made to the API.