Retrieving FME Flow Job Logs Using the REST API V4

Introduction

The FMEFlowLogFileRetriever transformer was deprecated in FME 2024.1 and was then completely removed from the product in 2025.0. Thankfully, FME Flow users can achieve the same functionality that this transformer provided by using the HTTPCaller transformer in conjunction with the FME Flow REST API V4.

In this tutorial, we will start with a very basic demonstration of how a user can pass in a job ID value to a workspace and then receive the associated log file back (either streamed in the browser or as a downloadable file). Then, we will build on this concept to show how this functionality can be applied more practically by running a workspace that returns a table of recent Flow jobs, which will include links for each log that will open the file in a new browser tab.

Requirements

• FME Form
• FME Flow
• An FME Flow API Token with the necessary permissions (explained below)

Step-by-Step Instructions

This tutorial is broken down into three basic sections. First, we will walk through how to generate an API token in FME Flow. Next, we will build a basic workspace from scratch that will retrieve a single job log. Finally, we will make some small updates to the provided workspace to demonstrate a potential use case for retrieving log files via the REST API.

Part 1: Create an API Token

1. Log in to FME Flow and Create Token

To start, we will need an API token that will give us the proper permissions to access the job log files we need. From your FME Flow home page, navigate to User Settings in the top-right corner, and select Manage Tokens. Ensure the API Tokens tab is selected, then select Create.

Give your new token a name (e.g. ‘Job Log API Token’), a description, and set an appropriate expiration date. Under Permissions, navigate to Jobs, and select the Manage permissions option. Click Save, and then be sure to download your token and save it somewhere safe, as this will be your only opportunity to download it.

Part 2: Build the Basic Workspace

Now that our API token is ready to go, we can start to build our first workspace. 

1. Create a User Parameter

Create a new workspace in FME Workbench. In the Navigator window, right-click User Parameters, then select Manage User Parameters. We are going to create a new user parameter that will allow the user to pass in the job ID value for the log file they would like to retrieve.

In the User Parameters window, click the green + button in the top-left corner, then select Number and fill in the parameter information below:

• Parameter Identifier: JOB_ID
• Prompt: Job ID
• Published: Yes
• Required: Yes
• Number Configuration:
  • Lower Limit: Greater than value
  • Value: 0
  • Upper Limit: None
  • Numeric Precision: Integer

Once your parameter has been configured, click OK.

2. Construct an API Request

If you are using FME 2025.1 and older, please add a creator transformer to the canvas first, and then connect the HTTPCaller.

As of FME 2025.2, many transformers have been updated to not require input from the creator transformer. For a list of all the transformers with this improvement, please see Transformers with an Optional Input Port

Add an HTTPCaller to the workspace, and open the HTTPCaller parameters. Configure the HTTPCaller parameters:

• Request URL: http://<Your_Flow_URL>/fmeapiv4/jobs/$(JOB_ID)/log
  • Replace ‘<Your_Flow_URL>’ with the URL for your Flow environment. Note that you may have to change the above request to HTTPS based on your configuration. This request passes in our user parameter as the job ID value, and will return the associated log file.
• HTTP Method: GET
• Headers:
  • Name: Authorization
    • Value: fmetoken token=<Your_API_Token>
  • Name: Accept
    • Value: text/plain
• Response Handling:
  • Save Response Body To: Attribute
  • Response Body Attribute: _response_body

Be sure to replace ‘<Your_API_Token>’ with the token value you downloaded at the beginning of this tutorial. Once you have configured all the necessary parameters, we can test that our HTTP request is working properly by clicking the Send Test Request button. Enter a valid job ID value for our JOB_ID parameter at the top of the window, then click Send Test Request. The preview window should then display the log file for the associated job in plain text format.

Once you have confirmed that the request is working, click OK on the request preview window and tten OK again on the parameters window to close it. If your request is not working, double-check your request URL and token permissions.

3. Write Log to a Text File 

Now that we have our log file information, we can write it to a text file. Add a Text File writer to the canvas, then specify a name for the resulting file (e.g., jobLog.txt). Click OK, then connect it to the HTTPCaller's Output port.

Double-click the writer to open the Feature Type Parameters window. Select User Attributes and set the value of text_line_data to the attribute that stores our log file response: _response_body.

Click OK to close the window, then run the workspace to ensure that there are no errors. Your workspace should now look like this:

4. Publish to FME Flow

Publish your workspace to FME Flow. Select a repository of your choice, and name the workspace GetJobLog.fmw. Register the workspace with the Data Download, Data Streaming, and Job Submitter services, then select Publish.

When we run the workspace from FME Flow, we will either see the log file displayed in the browser (Data Streaming), or we will be presented with a link to download the log file (Data Download).

Part 3: Complete Demo Workspace

Now that we have seen how to use the Flow REST API to retrieve job logs, we will demonstrate more practically how to integrate API requests into workspaces.

Download the GetMultipleLogs.fmw workspace from the ‘Files’ section of this article. This workspace uses two separate API requests:

The first returns a list of jobs based on two user parameters: the status of the jobs they would like to return (success or failure), and the number of jobs they want. These parameters are passed into the request to generate a list of recent jobs.

The second request is actually found in the AttributeCreator transformer, where we generate a new attribute for each job that is returned. This attribute contains an API request for the job log, which is wrapped in an HTML tag that generates a link in our list of jobs. When the user clicks that link, it triggers that API request and opens the log file in a new browser tab.

This second request will expose your API token in the URL of your request, which is not recommended for production or retrieving sensitive data. This is purely meant to be a demonstration of the different ways API requests can be made and how they can be integrated into workspaces.

1. Update the HTTPCaller

Open the workspace in FME Workbench. Open the HTTPCaller parameters and change the ‘Your_Flow_URL’ value in the request URL to your actual Flow URL. Update the ‘Your_API_Token’ value in the Authorization header to your actual API token.

2. Update the AttributeCreator

Open the AttributeCreator parameters and open the text editor for the log_request value by clicking the ellipsis button. Similarly, replace ‘Your_Flow_URL’ and ‘Your_API_Token’ with your actual Flow URL and token value, respectively.

For steps 1 and 2, note that you may have to update the request URLS to HTTPS, depending on your Flow configuration.

3. Inspect the Remaining Transformers

Open the HTMLReportGenerator parameters and click on the Table in the Page Contents bar. Here, you will see how we are populating the table with the log_request value for each returned job, along with some additional information, which we extracted from our first request using the JSONFragmenter (open that transformer as well to see how the request was queried to extract that information).

4. Publish to FME Flow

Publish the workspace to FME Flow in a repository of your choice. Register the workspace with the Data Streaming and Job Submitter services.

When we run the workspace, we can choose between returning successful or failed jobs, determine the number of jobs we want to return, and stream our HTML report to the browser. When we select the link of the job we are interested in, the associated log file will open in a new browser tab.

Additional Resources

HTTP Requests with the HTTPCaller

Getting Started with the FME Flow REST API