How to extract and use log information in Workbench

Use the Python FMELogFile object

The FMEObjects Python API is available for use in the Startup and Shutdown Python (Workspace Parameters - Advanced in Workbench), as well as in the PythonCaller and PythonCreator transformers. One of the objects available is the FMELogFile, which provides access to the Workbench log. Most of the FMELogFile methods are for adding messages to the log, but the setCallBack method allows you to send a copy of all log messages to your own function. This lets you parse the log messages as they are created and pass information from those messages to features in the workspace while it is running.

The three examples below illustrate some of the possible uses of log information during the workspace run.

Use Cases

Promoting warnings to failures

A customer found that some of his Microstation DGN files had been truncated. Normally, FME just posts a warning in the log file if it encounters a truncated file, but doesn't fail the translation. The customer wanted failures in order to separate the truncated files from the correct ones.

• The Startup Python in the FailTruncated workspace searches each log message for the DGN file truncated warning and sets a flag if found.
• A Creator transformer creates a new feature at the end of the process, after all the DGN features have been read.
• This new feature is sent to a PythonCaller, whose code checks to see if the truncation flag is set. If so, the workspace is terminated.
• The output of the PythonCaller is routed to a NULL writer, which is set as the first writer in the workspace.
• The second place DGN writer will not write its features until the workspace is complete, so no output file will be created if the workspace is terminated.

Extract File or Table Metadata

The Workbench readers often report file or table metadata to the log, but do not add this information to the features read from the file. The DGNMetadata example extracts the Global Origin, Master and Sub unit names and ratios from all the Microstation DGN files in a folder and outputs this information to an Excel file.

• The Startup Python in the workspace uses Regular Expressions to extract the Global Origin and Working Unit parameters from the reader log messages and store them to global lists.
• A single feature from each DGN file is read (by setting Reader Parameters - Advanced - Max Features to Read to 1).
• A FeatureHolder is used to ensure that all DGN files are read before continuing the process.
• The PythonCaller joins each feature to its metadata using the multi_reader_id generic attribute. This attribute contains the read order of each file, and corresponds to the index of the metadata Python lists.
• The attributes are renamed for a more pleasant appearance, then the features are written to Excel.

Output:

Save Intermediate Transformer Parameters

The AffineWarper transformer calculates a best fit Affine transformation from input control vectors, then uses it to transform the observed features for output. The calculated parameters are logged, but not stored on the transformed features. The GetAffineParameters example extracts those parameters into a CSV file for alternative uses.

• The source vectors are read from a CSV file and turned into vectors
• The vectors are sent to the AffineWarper and used to transform a dummy point
• The Startup Python in the workspace uses Regular Expressions to extract the affine parameters from the AffineWarper log message
• The Creator_2 transformer creates a new feature at the end of the process, after the warping is complete
• The PythonCaller adds the affine parameters to the feature, which is then written to CSV