Send Email Notification on Failed or Cancelled Jobs

Kezia Yu
Kezia Yu
  • Updated

FME Version

Introduction

In this tutorial, we’ll explore sending out email notifications for any failed jobs on FME Flow. To do so, we’ll set up an automation workflow to watch for failed jobs on our FME Flow instance and notify relevant stakeholders of the failed jobs. The automation leverages the FME Flow System Events trigger to kick off the process and runs a workspace to gather the failed job’s information and attributes. The workspace will read in the JSON message generated from the event and write out the attributes to the Automations writer. This enables the data to be used in the downstream Email action. This tutorial can be set up for canceled jobs as well (See Part 1, step 5). 

Below is the workflow process and sample email body that can be configured to notify of job failures. Part 1 of the instructions will walk you through an overview of the automation workflow and Part 2 will go through how the workspace is created.

 

Automation: 

  1. System Events Trigger: watch for Error Message Logged
  2. Filter Action: filter message content containing “Translation failed”
  3. Workspace Action: JSON user parameter linked to Event Attributes> Event as JSON.
  4. Email Action: Linked to the Automations writer port

 

Workspace:

  1. Processes incoming JSON message from the “Event as JSON” attribute
  2. Extract failed job ID using string transformers
  3. Extract failed job attributes using FME Flow REST API
  4. Clean up attributes before writing out to the Automations writer

 

Sample Email Body:

 

Requirements

  • Access to an email with SMTP
  • Licensed FME Flow and FME Form 

 

Step-by-Step Instructions

Part 1: Use the FME Flow Project

1. Download and Import the FME Flow Project

From the Files section of this article, download the linked FME Flow project, Sample Notify Job Failure.fsproject (created in FME Flow 2024.0). Import the project file to FME Flow by selecting Projects > Manage Projects, and “Import”. 

 

2. Manage Imported Automation

From the FME Flow web interface, go to Automations > Manage Automations to open the JobFailureNotify automation. The automation starts with the FME Flow System Events trigger to watch for the “Error Message Logged” events. These events can be viewed from the System Configuration > System Events page. 

From the trigger’s success port, we have the Filter Message action to filter the Message Content that contains the “Translation failed. Any error message logged in System Events that contains “Translation failed” will be passed to the success port to be processed by the workspace. 




3. Edit Automation 

On the automations canvas, edit the following nodes:

JobFailureNotify workspace action user parameters:

Ensure the JSON Message user parameter is set to “Event as JSON”. You can use the drop-down arrow next to the JSON Message parameter input and select Event Attributes > Event as JSON.

 

Set the admin username and password parameters. Click Apply and save the changes. 

Configure the Email action in the “Failed Job Notification” bookmark. 

Right-click the Email action and select “Enable”. The Email action will turn blue when enabled. 

 

Double-click the Email action and fill in the parameters. Input your SMTP Server domain name and set the “Email To” and “Email From” parameters. Notice the Email action input port is linked to the Attributes port of the JobNotifyFailure workspace action node. The Automations writer port enables workspaces to route data throughout the automation. This allows us to gather more data on the failed job and send the attributes to the next downstream action. You can also edit the sample email body and make use of automation parameters and attributes to include more detailed descriptions of the failed job.

Hi Team,

Please be advised that a Job ID: {route.Attributes.id} has failed on {route.Attributes.engineHost} FME Flow instance.
Please find the details of the issue below:
Job ID: {route.Attributes.id}
Workspace: {route.Attributes.workspace}
Repository: {route.Attributes.repository}
Message: {route.Attributes.message}
Engine Name: {route.Attributes.engineName}
Status: {route.Attributes.status}
Result: {route.Attributes.result}
Number of Errors: {route.Attributes.numErrors}
Number of Warnings: {route.Attributes.numWarnings}
Time Submitted: {route.Attributes.timeSubmitted}
Time Started: {route.Attributes.timeStarted}
Time: Finished: {route.Attributes.timeFinished}
Ran by username: {route.Attributes.userName}

Next Steps:
Please review the job log to initiate an investigation.


Kind Regards,
FME Flow Admin

Sent via Automation {automation.name} on {route.Attributes.engineHost}

 

Click Apply and save the changes.

Repeat step b. For the “Failed Automation Notification” bookmark.

 

4. Save and Start Automation 

Save and click “Debug” to start the automation in debug mode. To view the automation log, from the automation’s Menu drop-down, select “View Debug Log”. Clicking the Start button will run the automation in production mode. You'll find the log action messages in Resources > Logs > automations > current|old > action_logger.log. 

 

5. [Optional for Canceled Jobs Notification] 

For canceled jobs, we can edit/add the Filter Message action to filter for “Error Message Logged” that contains “Canceled” instead of “Translation failed.” 

 

Part 2. Authoring the Workspace

 

 

In this section of the tutorial, we will go through how the workspace works. The JobFailureNotify workspace can be broken up into 4 main sections (outlined using bookmarks). Follow the steps below to go through each section. The workspace is complete and configured but you may follow along by downloading the workspace (JobFailureNotify.fmw) file from the files section. 

 

1. Process JSON Message Content

The workspace starts with the Creator transformer and connects to the ParameterFetcher, where we can grab the incoming JSON message and assign it to an attribute. 

 

The incoming JSON message from the FME Flow System Events trigger looks something like this:

{"automation.id":"5a727ad7-0e38-4ef3-b7ee-4f4fd7dfb769","systemEvent.description":"Triggered whenever an error message is logged to fmeserver.log.","source":"system-event","time":"2024-01-12T09:39:33-08:00","message":"Job 15: Translation failed: \"Terminator: Termination Message: 'Unable to retrieve backup from FME Flow `https://<hostname>:443'. Please ensure that the credentials are correct and that user `admin' has fmesuperuser role. The error was: HTTP/1.1 401.'\". Job details are at https://<hostname>:443/fmeserver/#/job/15/summary","event.id":"08df5307-08c5-417f-8c64-f67bbfe2d5f0","automation.name":"JobFailureNotify [Framing](1)","systemEvent.title":"Error Message Logged"}. 

 

After assigning the JSON message to an attribute, we can use the JSONFlatterner transformer to parse the data. In the JSONFlatterner transformer, use the JSONMessage attribute as the JSON document and select the Attributes to Expose. 

 

2. Extract Failed Job ID

Once we have the attributes that we’re looking for, we can use string manipulation transformers to grab the job ID of the failed job. In the StringConcatenator transformer, we will use Regex to parse out the job ID and write to a new attribute, _jobID. 

  

 

With the SubstringExtractor transformer, we will further isolate the job ID.

 

With the StringReplacer transformer, we can remove any spaces in the jobID string value and replace them with a blank.

 

3. Extract Failed Job Attributes 

With FME Flow REST API, we can use the job ID to request more information on the failed job. To retrieve the job record according to the specified ID, we can use the following request:  GET /transformations/jobs/id/{jobid}. http://<hostname>/fmerest/v3/transformations/jobs/id/{jobid} 

With the HTTPCaller transformer, we can paste in the request URL, set the HTTP Method, and authenticate accordingly. We can save the response body as an attribute for further processing.

 

4. Expose Attributes

With the JSONFlattener transformer, we can expose the attributes from the response body.

 

5. Remove Unwanted Attribute and Write to the Automations Writer

Before writing the job attributes to the Automations writer, we can clean up the attributes with the AttributeManager transformer. 

In the AttributeManager, remove the following attributes: _creation_isntanace, _JSONMessage, _jobID, JobID, _response_body. 

Note: You may also use the AttributeRemover transformer here. 

 

We can now add the FME Flow Automations writer to the canvas and set the Feature Type Name parameter as Attributes. Ensure the User Attributes definition is set as Automatic. 

 

The workspace can be published to FME Flow and used in your automation workflows. You may go back to Part 1 to review how to create the automation. 

 

Troubleshooting

“My Automation is stuck in an endless loop” 

This may be because the JSON Message user parameter from the JobFailureNotify workspace is not set to the Event as JSON attribute. To do so, see Part 1, step 3a. 

 

“I’m not receiving the email notification”

Please ensure that your Email Actions are configured and validated. See FME Flow Troubleshooting: Email for more troubleshooting steps. 

 

"I imported the project (.fsproject) file but cannot view or modify the project's contents"

Please note that if you are importing the project as a non-admin user that you will need to have an Admin grant permissions to the project and contents before you can proceed. Please see Known Issue: Importing a fsproject File Requires an Admin to Grant Permissions to the Project for more information. 

 

Additional Resources

Routing Data Between Workspaces in Automations

Getting Started with Automations

Send Email from Automations

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.