Command Line Tools Integration

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2016.x

Introduction

A powerful pair - FeatureReader and FeatureWriter (new in FME 2016) - allows a simple integration of third-party command line tools into FME.

Any tool, any library that is able to read, process and save data in formats supported by FME can become a part of an FME workflow. For better usability, such external tools can be wrapped up into a custom transformer, so a user deals only with a familiar FME interface with no need to know any of the complicated syntax details a tool requires.

A general workflow for such an integration looks as follows. Note that most components depending on the workflow can be skipped (shown in blue). The only required parts are assembling a command line and executing it (shown in red).

7steps.png

Here are the three most common scenarios:

  1. Custom Reader transformer. A command line tool can read some format not supported by FME. The custom reader collects parameters from a user, makes a command line, executes it, and reads features with FeatureReader. Then, the features leave the transformer as if they were read with any standard reader. Users can then transform it and write it out to any of FME’s 350+ supported formats. See SVGReader example below.
  2. Custom Writer transformer. A command line tool can write a format not supported by FME. After gathering the parameters and saving FME features to a temp file, the transformer assembles a command line and executes it on the temp file(s). The results go to a user-specified location, as they do with any normal FME writer. See AnimatedGIFWriter example.
  3. Custom processing transformer. A command line tool can transform data in a way not available through standard FME transformers. With parameters set by the user, the data is saved to a temporary location with FeatureWriter, then the transformer makes a command line, executes it, and reads the processed features with FeatureReader. See LAStools_lasground_Caller example.

Components of the Integration Workflow

Each of the steps in the integration workflow diagram above are described here.

1) Gathering parameters

When a command line interface is replaced with a transformer GUI, it makes sense to create parameters that look user-friendly: instead of short options, use their expanded descriptions. A “Choice with Alias” user parameter allows showing such descriptions while the parameters get values that can be used directly in the command line.

Here’s how such a parameter may look within a command line:

os-saO31lV4QaFMGyAYGvZt06E_JxWHZ0wNXLeZnlkIbnuXn7zZ66LdUzzGbuDfr7isyXRyR1tafcxEmRAkJsvQ8EuEiOjxLQfGqYSbXw8Wci-FkMQEMkpdP7yq6ubwqWzJdGN9B

Within the Edit Choice dialog:

GEcOOW8BcnKvouwvwjjym8Ef9rPujik-wGPFb4ZmvATk30YzCdcemlTw3h23TJyU19dTtUISSIhQ0jIHc0rCP60-sRIkmOyPJJpR8GF6Po7IGiQpSFX7tgIMRCy7LQUIF3aw25Q6

and in the transformer’s GUI exposed to the user:

3EfkQN5am_ThGb3UKDTaVxCn_ju3I96u4zy0PE3dwVwQB3od_6uaBKzUfL4dNzSX0omuBWEapvwBpF0eKh5t8C5QUddSOoEnQlKEMdPyogKtPtsuaihfvy8FgR3zHoiHzNBOlISj

Some parameters may need additional processing. For example, we may need to recalculate a normalized color set through the FME “Color Picker” (for example, “1,0.5,0.25”) to RGB24 representation (255,127,63). In this case, the parameter can be fetched with ParameterFetcher, split into R,G, and B parts with AttributeSplitter, and assembled into required string with AttributeCreator. See SVGReader below for more details.

The path to the external tool may or may not need to be published as a parameter. For using a custom transformer within an organization, it might be better to hide the path from inexperienced users within the command line of the transformer. An FME professional will have to take care of setting the transformer up. If a tool is going to be shared across organizations (through FME Store, for example), it is better to publish the path:

40vOuVfFw5Zu8qNaTiXLEhIgo0rfxGrBAv10dZP72nc5M1ahZjkRG21dad8Ht7aeLQc8AXDRoepJxH8IcXW8yuN8vq8TRxpXgHTgny7Y5HnC0VZe5DvGySo2YOWwIke9dTLcdlAS

 

2) Reserve temporary file names

While it is possible to generate unique names for temporary input and output files using regular FME transformers, it is more reliable to use a special TCL command within TCLCaller transformer. The syntax of the command looks as follows:

FME_TempFilename <file_name_prefix> <file_name_suffix>

It creates an empty file with a unique name in the FME TEMP folder.

For example, if we would like to make a temp file for using one of the ImageMagick utilities, we can write the following TCL expression:

O9KypSButla5k4VHBBVC9DIBcQYK24nJMnEUEQ0kIbDnS35VXWwLjvDKUJAPM8-YAvstYTAPsrHSXJxE_PyHbpogqP658B66OWDlVjJPyQ8TIHIGeBmMTEyWJQAEvoPo6IB6TQGv

The expression will generate a name similar to this:

C:\Users\username\AppData\Local\Temp\ImageMagick_1453409310204_7104.png

Now this name can be used for making a command line, setting as transformer parameter and filling the file with some contents.

 

3) Save data with FeatureWriter

Depending on a workflow, it may be necessary to save the intermediate files to some temporary location or use the source files directly. For example, when we use LAStools for ground classification, we may first clip the area of interest and classify only this part of the original point cloud. Another example is when the source format is not supported by LAStools. In these cases we need to make the temporary LAS files.

Using the file name reserved with TCL command (see above) and FilenamePartExtractor transformer, we can create attributes specifying path to the temp output dataset and the root name for the file:

MtU1ZF8XfgDu4UVQiBG2ZLI3H33sQFQYehCdEOc9OfvOqYo4sxpTm3EeXjcN47LPnt770hPHXeSu2zEvVmHAdraIqDEaLxt6HjBIp0uyBrKeu7WdMebKeMFwoeCZ8Xm_hezrqZ6G

After that, it’s easy to use these attributes within FeatureWriter:

07i7zNndvjHreXSJ_hPhD9dwPrbQkVM3a-hOQZLYtIfckqJ04JW1aWUh0GJk495Kp8G3oOzGS8feZWIkG1IvfGyTLs-9MIhMXEU59gbe9E2zyxFCf2DpKF4FTaFyPxM6ED5y9hZq

 

4) Assemble and execute command line

The command line can be written directly in the SystemCaller transformer or made with AttributeCreator as an attribute, which then will be used in SystemCaller.

The command line should include:

  • the full path to the program,
  • all the user parameters either “as is”, for example, “$(DELAY)” or made into attributes with ParameterFetcher or AttributeCreator (@Value(_frame_delay)), and processed if necessary,
  • the path to the source data - temporary or original,
  • the path to the destination data - temporary or final.

Here is a couple of examples:

“"$(IMAGEMAGICKCONVERT)" -delay $(DELAY) @Value(_ImageMagick_sequence_file) -loop $(LOOP) $(GIFOUTPUT)”

""$(LASTOOL)" -i "@Value(_input)" -o "@Value(_lastools_temp_output)" -scale_u $(HEIGHTSTORAGESCALE) $(HEIGHTSTORAGE)"

Always make sure the paths in the command as well as the whole command have double quotation marks around them - this helps to avoid potential problems with some characters in paths, such as spaces.

In some situations, the assembled command line may become too long exceeding the allowed limit. On Windows, the limit is 8192 characters. There are a few possible ways to work around this limitation. For example, instead of listing all the files in a command line, it is possible to use a wildcard, see AnimatedGIFWriter example. Or use shorter names for folders and files by choosing FME_TEMP folder location closer to the root. For example, instead of C:\Users\username\AppData\Local\Temp\, use E:\FME_TEMP\

The assembled command line attribute goes to SystemCaller, which executes the process and waits for results before continuing with other parts of the workspace. The external tool may also report back to the log window the status of the job or error messages.

 

5) Read Data back with FeatureReader

Once the external tool finishes its job and makes some output, FME can bring the output back with FeatureReader for further processing. In some cases, it might be necessary to merge the processed data with the original.

For example, the point clouds in formats not supported by LAStools may need joining with the processed data for preserving the original components. This can be done with PointCloudMerger by X, Y, and Z of both the source and processed point clouds.

 

6) Remove temp files

After the external process finishes, and the data is brought back to the normal FME workflow, we may need to remove the temporary files. It is possible through the OS commands, such as “del” in SystemCaller, for example:

del "@Value(_lastools_temp_input)" /Q

In some situations when dealing with point clouds and rasters, it is more reliable to issue the following FME command with FMEFunctionCaller:

@Geometry(DELETE_SOURCE_DATASET)

_b8oj5Vu-bO_ixaeIGVzxCCKEcEHqZWmU_uovfxuYa53XrfFH6JP0w-SYGu5e7Uzw7p45zoXS7RzBR6Fq6HI6itiIi6Z11JBeuzzQiOmcydeTVXksxaHI7OhPRzIFpjgmFOR3SUm

This transformer can be placed right after reading the geometry or later - in any place where the geometry of the feature still exists. The transformer removes the temp file as soon as it is not needed anymore-for example, if its geometry is destroyed in the FME workflow or the translation ends.

Note that both OS “del” command and the @Geometry(DELETE_SOURCE_DATASET) function should be used with caution to avoid deleting the original source data or temporary files before or during their use by an external process.

 

Note on Customization

The general integration workflow described in this article does not have to be placed into a custom transformer. Creating such a transformer, however, makes it easier to separate the task accomplished by an external tool from the main workspace. It is easier to debug, update or modify a single custom transformer than the whole workspace, which can be really big and complex. All the examples shown here can easily be modified by right-clicking the custom transformer and choosing “Edit”. The transformer will open in a new Workbench tab. Learn how it works and feel free to change it the way you need it - add or remove options, change the tool performing the transformation, translate the parameters into a different language, or chain a few tools together into a single transformer if this makes sense in your working environment.

 

Examples

Example 1: SVGReader

SVG (Scalable Vector Graphics) is an XML-based vector image format for two-dimensional graphics with support for interactivity and animation. FME supports writing SVG files, but not reading. We can read SVG files as rasters using ImageMagick, a free and open-source software suite for displaying, converting, and editing raster image and vector image files. It can read and write over 200 image file formats.

Note that ImageMagick is not a part of FME. You can download your copy of the product from here - http://www.imagemagick.org/

The SVGReader custom transformer allows reading one or many svg files and bringing them into FME as rasters with FeatureReader. The raster(s) then can be stored in any supported raster format. Note that some formats may require adjusting their raster interpretation (See RasterInterpretationCoercer). FeatureWriter is not used in this custom transformer.

It is also worth noting how we set the color and transparency of the background - the parameter is set as a normalized FME color, and then changed to ImageMagick rgba model with AttributeSplitter and AttributeCreator:

"rgba(@Evaluate(@floor(@Value(_color{0})*255)),@Evaluate(@floor(@Value(_color{1})*255)),@Evaluate(@floor(@Value(_color{2})*255)),$(ALPHA))"

SujdUekisiLJkiUB8LQZnmWNXRNg4usR3Tao-18WZn3qF4MxNXjZ5scddgYLNzx4DoT0mpJ8I9G-h-BXprgnRvNVQqJTKvZvpiJAVcczyO7b2zMdQPeQOPDvDyXMiYjE0lyKpCcN

Download the template:svgreader.fmwt

 

Example 2: AnimatedGIFWriter

Another example using ImageMagick. AnimatedGIFWriter expands FME’s writing capabilities by adding animated GIF support.

Using FME allows much easier creation of the georeferenced animations, for example, driving maps.

The workspace example shows how to pass the background extents to the moving parts of the animation so a user does not have to calculate the location of the rasterized features in the animation.

The custom transformer uses FeatureWriter to save the background and all the animation frames into temp PNG files. ImageMagick assembles them into the GIF.

5ru0reL6WJRQ1YNkILvWs7kHt_gGFr1VnDcyYxgwrq-OBzjg1TSeu_9jLG5TEYDEpCOddER1A51CC0yMj69SDoVBthh-Ev-tBGafCWhBexBSPNs9r1cKjmtaD4seCZWun9YT82ZL

FME 2016 does not support animated GIFs, so it makes no sense to read the file back and display it in Data Inspector. The easiest way to see the animation is through a web browser:

gst98nlZP8LgrrzMCmn3tlDx0N83z3_h43jTsANHXsLX5RYzZtGCRbNV6NttuvfIFxjtI_bPUEOyPOoZAxzGEzA-MVGfAqMMAChu5f7dt5JvpNQVKY9XNDPhe2TfsPuC-s3SoIpl

Download the template: work2home.fmwt

 

Example 3: LAStools_lasground_Caller

The popular LAStools software suite is widely used by many organizations for processing LiDAR data. For FME 2016, we made several custom transformers that wrap up a few LAStools instruments.

Note that LAStools suite is not a part of FME. You can download your copy of LAStools from http://rapidlasso.com/. For pricing and licensing information, visit http://rapidlasso.com/pricing/ and http://www.cs.unc.edu/~isenburg/lastools/LICENSE.txt.

LAStools_lasground_Caller calls lasground.exe, which classifies points as ground and non-ground.

CESZqP4EB9l2Sr_bH5CSspCjT-I04YmYuwL-B6KnkCGVl0PBvkpS9hIm30Om0PhtvjfZZpl36OFUDTeDhYkxqleP4BDSfRBsLMX97yjqNptjI3odTMyvP5IzvfYTTbPCMheJsrhW

In this example, FeatureWriter is used when the source file was changed before entering the transformer, or the source point cloud format is not supported by LAStools. In these two cases, the temporary LAS files are created in the TEMP folder and lasground.exe runs on these files, otherwise it works directly with the source files:

UxOrWw8GIjVGvsspc_OHsLWRWB_2eUrcYhstNfo5iSobtYOH0UEGtAQ8iIBdSHfYMX1JUzXa2hICuO1aJug6jrmcAhS_srvTD459dAhwatT-E2yGkeH2M4Cj6LEYBNP0yqa_WNRK

FeatureReader is always used to bring the results generated with lasground.exe back to FME.

An interesting part of this workflow is merging the original point cloud with the generated LAS files. If the source has components that are not supported by LAS, they will be lost after lasground.exe finishes its job. PointCloudMerger allows combining the original point cloud with the results by X, Y, and Z:

ppe68wsd9brtWH5_h3EcuUVFZFrXhQ9NB77sm9PSdoPSRRdyKrBxduOkE3rB_jI5V_N4BmxUJ-000TeLHeP2ZhZLSo9csA8T4EUtMhyFQ3HOf4GPI_vUm3jAteVm_pAeU902ixgE

Note that merging by XYZ will not work if a tool changes one of these components.

Download LAStools wrapper transformers: lastoolstransformers.zip

Download the template (contains the workspace with 5 LAStools wrapper transformers):lastoolsdemo.fmwt

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.