Monitoring FME Server Job Activity using the REST API

Introduction 

Monitoring the activity of FME Server enables you to make more informed decisions about how you would like to respond to increased demand.  

In FME Server 2021.0+ additional job statistics have been exposed to show job performance. These metrics are available to view through the Web UI on the Jobs Completed, Workspace, Automations, and Schedule pages as well as accessible via the REST API. 

This tutorial will demonstrate how to use the REST API to retrieve these statistics and build a visual report that can be displayed in FME Server Dashboards to help monitor FME Server activity, performance, or usage over time. First, let’s take a look at what these new statistics are!

Job Statistics Overview

Statistics Available Per Job

Elapsed Time: Total processing time for an FME Workspace (End Time - Start Time). 

CPU Time: Total CPU time for an FME Workspace (User time + System time). 

Note: CPU Time is time spent doing work on the CPU (as opposed to waiting for input, disk, network, etc.).
CPU Time can be further subdivided between User (FME doing work) and System (the OS doing work on FME's behalf).

You may observe discrepancies between the CPU time recorded in an FME Desktop/Server job log and the time presented through Job Statistics (on the Jobs Completed page or via the REST API). This is expected because the time reported by Job Statistics includes the additional start and end scripting that cannot be captured by FME Desktop/Server logging processes.

CPU Percent: The percentage of total processing time that is recorded as CPU time (CPU Time / Elapsed Time). 

Peak Memory Usage: The net memory usage, is calculated as the peak memory used by an FME Engine process at the end of translation subtracting the memory in use by the FME Engine before the translation begins.

Note: Unlike in FME Desktop the FME Server Engine processes do not shut down after every job and this can lead to unintentional memory leaks between translations. To account for this accumulation and return a more accurate value for peak memory consumption the above calculation is performed. This can result in different values presented through Job Statistics (on the Jobs Completed page or via the REST API) and in the FME Desktop/Server log files. The latter shows only the peak memory used by the FME Engine process at any time during the translation, without taking into consideration the memory already in use at the start.


Jobs Completed Page with statistics columns

To fetch this information through the REST API use:

All Jobs [GET] /fmerest/v3/transformations/jobs/completed

Per Job [GET] /fmerest/v3/transformations/jobs/id/< jobid >

Statistics Available Per Workspace, Schedule or Automation

Total Runs:  The number of times that the workspace has been run. 

Average Elapsed Time: The average elapsed time across all runs. 

Average CPU Time: The average CPU time across all invocations.

Average CPU Percent:  The average CPU percent across all invocations.

Average Peak Memory Usage: The average amount of memory an FME Workspace required while it was processing. This is the average of the Peak Memory Usage above.

Workspaces page with statistics columns

Schedules page with statistics columns


Automations ‘Run a workspace’ Action with statistics columns

To fetch this information through the REST API use: 

All Workspaces by Repository [GET] /fmeserver/v3/repositories/<repo>/items  

Per Workspace [GET] /fmeserver/v3/repositories/<repo>/items/<item/workspace>

All Schedules [GET] /schedules

Per Schedule [GET] /schedules/< category >/< name >

Per Automation It is not currently possible to retrieve this information via the REST API. 

Note: If you would like to reset the statistics, this is an option per workspace or schedule through the Web UI. For Automations, statistics are reset automatically whenever you stop and restart the Automation. 
 

Use Cases

Dynamic Engines

As opposed to paying for a fixed license per engine, Dynamic Engines run on credits giving you the option to purchase additional FME Server processing power in a whole new way. One credit equals one hour of engine CPU time. To understand when it might be appropriate to route jobs to dynamic engines, administrators need to have a good understanding of the average CPU time their workspaces are using. Check out the Getting started with Dynamic Engines article to learn more about this feature. 

Queue Control 

FME Server Queues have been rewritten in 2021.0+ to provide users the ability to configure advanced job routing rules enabling workspaces to be directed to engines based on how they have previously performed according to average job statistics. In order to make informed decisions about how to configure job routing rules an understanding of how these jobs are performing is required. 

Schedules

By understanding how long a workspace configured on a schedule is taking to run on average, this can help FME Server users set schedules to help avoid long queue wait times. 

Managing Hardware

Your FME Server architecture may include engines distributed across hosts with different hardware specifications. By understanding the peak memory usage of different workspaces you can route jobs accordingly to improve performance and even look into autoscaling of new compute resources. 

Performance Tuning

By understanding when your engines are most or least busy and how much time is spent performing certain tasks this information may help to create and modify workflows to utilize engines more efficiently.

Auto-Scaling Engines

This lays the groundwork for autoscaling to enable organizations can optimize how engines are deployed so watch this space!
 

Step-by-step Instructions

The following example uses a pre-made workspace that will be published to FME Server to generate a dashboard. The dashboard will use a third-party tool Plotly.js to generate a Box Plot showing the peak memory usage per job, and information that has been retrieved from the jobs/completed endpoint.  If you want to build your own workspace from scratch check out the Tips for Working with the FME REST API in FME Workbench article for some things to look at whilst getting started. 

Note: The attached example workspaces (FMEServerAPI_JobStatistics_Dashboards_Workspaces.zip) provided require FME 2021.0 or newer to run successfully. 

1. Open Starting Workspace
In FME Workbench, open PeakMemUsage_BoxPlot.fmw starting workspace, which is available for download from the Files section of this article. 

2. Explore the Workspace
Explore the workspace and make sure you understand what each step is doing. Set the AUTH_NAMED_CONNECTION published parameter value to your FME Server Web Connection. Note: To run the workspace on FME Desktop you'll also need to set the value for the FME Server Parameter FME_SERVER_WEB_URL to your FME Server e.g. http(s)://<hostname>:<port> and the FME_SHARED_RESOURCE_DASHBOARD parameter to a file path to save the output. 

3. Publish to FME Server

Publish the workspace to the Dashboards Repository (You may need to ask your Admin to update your user to have write permissions to this repository).

If not already on FME Server, upload the FME Server Web Connection

Register the workspace to the Job Submitter and Data Streaming Service. This way the workspace can optionally be run on the fly and stream the dashboard directly back to the user who requested the information. 

4. Run Workspace 
To confirm everything is working correctly run the workspace in FME Server

5. View Dashboard
Go to Jobs > Dashboards and from the Name select PeakMemoryUsage. This is the name of the HTML file that this workspace creates and can be modified in the Writer dataset in FME Workbench. 

The third-party tool plotly used to generate this workspace has an interactive interface. Explore your result by highlighting a section to zoom in, or hover over a box/point to get additional details about the job statistics. 

6. Configure Schedule

 For the dashboard to remain up to date and show the latest statistics the workspace must be run at a repeat interval. Select Schedules > Build Schedule

Specify the interval at which you want the workspace to run (i.e. how often the dashboard should be updated)
Now the dashboard is all set up and will keep up-to-date!  
 

Dashboard Examples

Attached to this article are workspaces for a series of more simplistic job statistic reports that make use of the HTMLReportGenerator. To set these up on your FME Server, follow the instructions above for each workspace. All the workspaces include minimal filtering options which can be used if a Workspace App is built to return the report on demand. Please add a comment below if there are any other dashboards you’d like to see!
 

Average CPU Time/CPU % and Peak Memory Usage Per Workspace

A basic bar chart to cover each new statistic using the /jobs/completed endpoint. Specify how many workspaces to return and a date range (e.g. up to 7 days ago). Additionally, filter results for a single repository. 


Report generated from AverageStatistics_Basic.fmw
 

Job Statistics Per Repository Overview

Dashboard to display all the new job statistics using the /repositories/<repo>/items endpoint. A report has been created for each repository divided by a tab. Each report contains a bar graph displaying the average CPU vs elapsed time per workspace, and underneath the table lists CPU percentage and peak memory usage, as well as how many job runs this information is based on. Optionally filter by Repository name. Results cover the period since the statistics were last reset. 


Report generated from JobStatisticsPerRepository_Chart_Table.fmw
 

Workspace Status per User

As a user perhaps you want to perform a health check of all of the jobs you own, or perhaps as an administrator you’d like to see how active each of your FME Server users are. This workspace makes use of the /jobs/completed endpoint to create a report of all the workspaces belonging to a user and how much they’ve been run, sorted by the job status.


Report generated from WorkspacePerUser_BarChart.fmw

Troubleshooting

If you are experiencing issues please consult our FME Server Dashboards troubleshooting guide, if this does not resolve your problem then contact Safe Software Support.
​​​​​

Additional Resources

Documentation: FME Server Dashboards
Tips for Working With the FME Server REST API in FME Workbench

Data Attribution

Plotly.js is free and open-source software, licensed under the MIT license.