Getting a Remote Engines Service Job Log from the FME Flow Core

Files

GetRemoteEngineJobLog1.2.0.fsproject
- 700 KB
- Download

Introduction

FME Flow Remote Engines Services offer several advantages, including the ability to deploy processing close to the data on any supported operating system without network complexity. The one-way architecture that makes this flexibility possible introduces some limitations compared to a standard distributed engine:

The FME Flow core job log only contains information about the remote engine job submission.
All job log details are stored locally in the remote engines service. They are not accessible from the FME Flow core installation UI.
Only the remote engines service administrator can access job log details to see if and why something went wrong.

This article presents a solution for retrieving a remote engine job log using several FME Flow tools:

A Workspace App for a user to enter a core job ID
A Data Virtualization API to verify the user’s job permissions and get the associated remote engines service URL and job ID
The FME Flow REST API to retrieve the job log from the remote engines service

The diagram below illustrates the flow:

Requirements

FME Flow 2026.1 or later
FME Flow Remote Engines Services connected to an FME Flow core installation
Administrator access to FME Flow and FME Flow Remote Engines Services

Step-by-Step Instructions

Part 1: Import the Project

A sample FME Flow project has been created and attached to this article. To learn more about importing projects, please refer to the FME Flow Documentation. The basic steps are below.

To follow along with this tutorial, please download the GetRemoteEngineJobLog1.2.0.fsproject file from the Files section to the right of this article’s content.

1. Import the Project

Log in to FME Flow as an administrator. Go to projects and click Import. Select the fsproject file you downloaded. Click Import.

2. Preview the Content

Provided you do not have items already in your FME Flow installation with the same names, everything should be imported as new items. The items are:

GetRemoteEngingLog Workspace App, along with its workspace and repository dependencies
Permissions Data Virtualization API

Click Continue.

3. Proceed with the Import

For the next step, “Choose Items”, click Continue again, as there are no selectable items.

For the “Resolve Owners” step, you’ll be prompted to choose a new owner for the project items. This project was created by a user named “dv_admin”. Choose the admin user you are currently logged in as.

Click Continue.

Finally, click Import Project to start the import.

4. Confirm That the Project Imported Successfully

You should receive confirmation that the project import was successful. If not, click View Details to review the log for error messages.

Part 2: Review and Configure the Workspace App

With the project imported successfully, let’s start reviewing the solution from the top down, beginning with the workspace app the user interacts with.

1. Open the GetRemoteEngineLog Workspace App

Go to Flow Apps, Workspace Apps, and click on GetRemoteEngineLog.

2. Review the App Details

As the description states, this app allows a user to enter a core job ID and obtain the associated job log from the Remote Engines Service.

To verify if a user has access to a job, authentication has been enabled. The default FME Flow roles that can access jobs are allowed, but this can be changed depending on your installation and role permissions.

Role-based access control (RBAC) is always strongly recommended when managing access. As an owner, dv_admin appears in the Allowed Users list, but all other access is managed through roles.

4. Configure the Parameter Defaults

Expand Parameter Defaults and update them as follows:

1. Core Job ID: This is the only parameter displayed to the user. Leave it as 1 or enter a different numeric value.

2. Core Admin Token: the API token used to check job permissions on the FME Flow core using the Permissions Data Virtualization API and FME Flow REST API. It must be a token created by an administrator user with ‘All Permissions’ enabled, including the new Permissions Data Virtualization API imported earlier. For example:

If you have not created an API Token before, please follow the steps here: Authorization in the FME Flow REST API | Token Management.

3. Remote Engine Administrator: Enter the admin credentials for your remote engines service. These are the same credentials used when adding the remote engines service in Engine Management. Note that at this time, only one set of remote engine service credentials is supported (if you have multiple). Also, remote engines services in Snowflake are not supported due to the extra layers of authentication required.

4. Verify Remote Engine SSL Certificate: If you’re receiving SSL errors, set this to No for troubleshooting. Otherwise, it should be set to Yes. If your remote engines service is not secured with a recognized CA cert, you may need to import the certificate into the core server before setting this to Yes.

5. Save

Save the changes you’ve made to the workspace app.

Before running the app, let’s understand the workspace it runs in and the data virtualization API it uses.

Part 3: Review the Workspace

1. Download the GetRELog.fmw Workspace

The workspace is located in the Remote Engine Tools repository, which was created when the project was imported.

2. Open the Workspace in FME Workbench

The workspace has three bookmarks:

Check job permissions

Get the log

Handle errors

We won’t review every transformer in each bookmark; we'll focus on the key ones.

3. Review the “Check job permissions” Bookmark

The “Check job permissions…” bookmark is the first light-blue bookmark on the top-right. The main consideration here is the HTTPCaller_GetJobPermissions.

The HTTPCaller_GetJobPermissions:

Makes a GET request to the data virtualization endpoint /permissions/jobs/{id}, using the FME_SERVER_WEB_URL FME Flow parameter and passing the Core Job ID the user provides as a parameter.
Supplies the user ID of the authenticated user who submitted the GetRemoteEngineLog Workspace App request via the FME Flow parameter FME_SERVER_RUNTIME_USER_ID
Authenticates with the admin token you set up in Part 2

This call returns information about the core job and its associated remote engine job, including the server URL and jobId, or an error if the user is unauthorized.

We’ll take a closer look at how the Permissions Data Virtualization API works in the next part. If the request succeeds and the user is authorized to view the requested job log, the response JSON is parsed, and we proceed to retrieve the job log from the remote engines service.

3. Review the “Get the log…” Bookmark

The “Get the log…” is the green bookmark on the top right. If the user has permission to view the core job log, then the HTTPCaller_GetJobLog will download the associated remote engines service job log.

The HTTPCaller_GetJobLog:

Makes a GET request to the remote engines service’s REST API endpoint /jobs/{id}/log, using the URL and JobID from the /permissions/jobs/{id} response
Authenticates with the remote engines service admin credentials, which are provided via parameters
Returns the job log as binary data and saves it to an attribute

The log file data is written to the FME Flow Shared Resource Temp location, $(FME_SHAREDRESOURCE_TEMP)/RemoteEngineTools/logs, using the AttributeFileWriter. A FileCopy writer is used simply to register with FME Flow’s Data Streaming service, so that the job log can be streamed back to the user.

4. Review the “Handle errors…” Bookmark

The last bookmark in the GetRELog.fmw workspace is the red “Handle errors…” bookmark. Any errors, such as unauthorized access, are logged here so they can be returned to the user as HTML.

The HTMLReportGenerator formats the HTTP status code and any error message into HTML.

The HTMLReportGenerator includes CSS styles to improve the appearance of the error message streamed back to the user. You can revise this style later, if you’d like.

In the screenshot, this transformer is incorrectly flagged as incomplete due to a small bug that was unresolved at the time this article was written. The error can be ignored.

Part 4: Review the Permissions Data Virtualization API

Now that we understand the app and workspace, let’s take a look at the Permissions Data Virtualization API endpoint that handles the job permissions check.

1. Understand Job Permissions

First, to understand the expected results, it’s important to understand job permissions in FME Flow. Two levels of job permissions can be assigned at the user or role level:

Access: access your jobs
Manage: access and manage all jobs

To learn more, please see the FME Flow documentation on permissions.

The Permissions Data Virtualization API makes several FME Flow REST API calls to get a user’s effective job permissions: the intersection between their assigned user and role permissions. If the user has access to a job on the core and that job has an associated remote engine job, they can, by extension, access the remote engine job log.

2. Open the Permissions API

Go to Data Virtualization, then click permissions.

3. Review the /jobs/{jobId} Endpoint

This API has a single endpoint: /jobs/{jobId}

For the purposes of this article, we’ll review the Swagger API documentation generated by data virtualization. If you want to learn more about data virtualization, please see Getting Started with Data Virtualization. If you’re already familiar with data virtualization, you can use FME Form to download the workspace used by this endpoint.

To view the Swagger documentation, click the API Details tab, then View Documentation.

The Swagger documentation tells you all about the /jobs/{jobId} endpoint, including the:

Type of method (GET) and the endpoint (/jobs/{jobId})
Parameters this endpoint requires
Potential responses and response schema

This information is used to create the request inside the HTTPCaller_GetJobPermissions transformer, which was reviewed with the GetRELog.fmw workspace in Part 3.

Part 5: Test the Solution

Let’s try getting a remote engine job log from the FME Flow core!

1. Submit a Job to the Remote Engine

Confirm that the job was submitted to the remote engine. Take note of the job ID.

2. Retrieve the Job Log

Open the GetRemoteEngineLog workspace app:

<your FME Flow URL>/fmeserver/apps/GetRemoteEngineLog

Enter the Job ID and click Run.

The workspace app calls the data virtualization endpoint to check permissions, and if it’s successful, streams the job log back. Your browser should begin downloading it automatically. Note that the ID for the job log downloaded will be different than the core job log ID you entered.

3. View Data Virtualization Jobs

Each time the workspace app runs, the Permissions Data Virtualization API is called, and its workspace is run. You’ll see this on the Jobs page in FME Flow:

4. Test an Error Message

Try some other use cases to further test the app.

Log out of FME Flow and log in to the GetRemoteEngineLog app as a user in the fmeauthor role. The fmeauthor role only has Access job permissions, so users in this role should not have access to the test job submitted in the last step.

Try to get the job log. Since the user doesn’t have access, you should receive a 403 error:

Try to get a job log for a job that was not submitted to the remote engine. You should receive a 422 error:

You now have a way for non-remote engine administrators to get their job logs. If you would like, you can return to the workspace app settings and customize the appearance.