Benchmarking FME Form

Introduction

In this article, we’ll go over what benchmarking is, and how it can be done for workspaces and transformers in FME Form. 

What does it mean to benchmark software?

Benchmarking software is the process of assessing the performance of a software system by running it against standardized tests. It serves to establish a set of metrics that can then be compared against older versions to evaluate performance gains, or to identify inefficiencies in your software or environment. 

At the moment, FME doesn’t come with a built-in way for users to benchmark transformers or workspaces, but we can use FME Form to create a simple test.

Additional Downloads

aggregator_benchmark.fmw

point-cloud_benchmark.fmw

The Workspaces

Two workspaces are required for this process: the benchmarking workspace which is attached to this article, and a test workspace. 

The Benchmark-Workspace.fmw workflow essentially runs the test workspace n amount of times (the number of times is specified by the user at runtime), and then retrieves the logfiles from those runs. From there, the benchmarking workspace extracts a couple of metrics from each logfile: 

• session_duration: The time it took for each test workspace to run, and
• cpu_user and cpu_system: The amount of user and system CPU that was used when running each test workspace. 

With the metrics in hand, we can then calculate the average, which will serve as the result of our benchmarking test. Lastly, an HTML report is created to graphically display the results. 

The Test Workspace

First, prepare a test workspace. This could be either a full workspace or a much simpler workspace that focuses on the transformer that you’re looking to benchmark. 

If you’re looking to test a specific transformer, the recommended structure would be to start with some sample data or a Creator, then the transformer to be tested. An example can be found in the attached aggregator_benchmark.fmw workspace. 

For a full workspace, preparation could be as easy as removing all the writers. An example of a workspace with more than one transformer, and that takes longer to run, can be found in the attached point-cloud_benchmark.fmw workspace. 

Whether you are testing a transformer or a full workspace, it’s important to note a couple of things:

• The test workspace shouldn’t include any writers, as this can cause problems when running it through the benchmarking workspace. Feel free to experiment with this, though.
• The test workspace should ideally be an .fmw and not an .fmwt. If you intend to use an .fmwt, make sure there are preexisting logfiles for the workspace to write to, since .fmwts don’t generate new logfiles on their own.
• Ensure the test workspace data is in the same location as the test workspace itself (subfolders are okay). 

Now that your test workspace is set up, it can be used on the benchmarking workspace.

The Benchmarking Workspace

When using the benchmarking workspace, the first thing you’ll need to do is ensure that Prompt for Parameters is enabled. This can be done via the drop-down arrow next to the Run button.

Now, when the workspace is run, the User Parameters window will pop up. Here’s where you can configure the settings for the run. 

The “Number of times to run the workspace” number specifies how many times the test workspace will be run. The higher the number, the more accurate the results will be but keep in mind that this will take longer and use more resources. 15 is the default. 

The other parameters are where you select your test workspace as well as a location and name for the HTML report. By default, the HTML report will be saved in the same location as your benchmarking workspace, and be named “Benchmarking-Results-<runtime>.html”.

Click the Run button on the User Parameters window to initiate the translation. When it's finished, view the output HTML report. Below is an example: 

Note that if you’re running the benchmarking workspace more than once, you’ll either have to delete the generated .log files every time, or you’ll have to specify a new folder every time to avoid having the FeatureReader read in all the same logfiles again.

What’s next? 

The benchmarking workspace in this article can be used as-is, but as with all FME workspaces, there’s always room for customization! Here are some suggestions: 

• Write to different outputs. Instead of creating an HTML report, the result can be emailed, written out to an Excel file, or sent to a dashboard.
• Add different messaging to your output: identify and extract WARN, STATS, or ERRORS in your logfile. The LogMessageStreamer can help with that.
• Auto-delete the test logfiles once the benchmarking workspace is complete. A TempPathnameCreator would be helpful here.
• Create a Flow app that allows for self-serve workspace benchmarking. The logfiles can be stored and then deleted from a Flow repository.

Now that you have a better understanding of what time and CPU resources your workspace uses, you can visit any of the Performance articles to further improve your workspace’s efficiency. The Performance Tuning FME article is a great place to start. In addition, your logfile can also help identify places in larger workflows where you’re experiencing memory slowdowns or inefficiencies in parts of your translation.