Integrating ArcGIS Enterprise Geoprocessing Webhooks With FME

Matt Meeboer
Matt Meeboer
  • Updated

FME Version

  • FME 2023.0

Introduction

Esri’s ArcGIS Server, which is included with ArcGIS Enterprise, offers webhooks that can send data to FME Flow when specific events occur in near real-time. As of ArcGIS Enterprise 11.1, Esri has introduced geoprocessing webhooks. 

A geoprocessing webhook will send information about a geoprocessing service when it completes on ArcGIS Server. For example, we may want FME Flow to start a workflow when a long-running geoprocess completes on ArcGIS Server. This may include:

  • Sending out a notification on geoprocessing success or failure
  • Initiating a process in another application
  • Initiating other downstream processes

Unlike organization or feature service webhooks, geoprocessing webhooks do not have specific event triggers that can be defined. The webhook will send data every time the subscribed geoprocessing service completes.

In the following tutorial, we will set up an FME Flow Automation to send an email every time the SpatialAnalysisTools geoprocessing service completes. This is a system service that comes with ArcGIS Server and enables basic spatial tasks , like creating buffers. This is a simple example for the purpose of this article. In reality, you may wish to monitor a custom geoprocessing service created from a model in ArcGIS Pro, for example.
 

Requirements 

 

Step-by-Step Instructions 

Part 1: Create the ArcGIS Enterprise Portal Webhook


1. Configure the Automation in FME Flow

Start by building a new Automation in FME Flow that will receive messages from the ArcGIS Enterprise Portal. 

Set the Trigger to Webhook (triggered), leave Required Authentication unchecked, and click Apply. Attach a Log a Message Action to the Action Succeeded port (the checkmark) of the Trigger. 

Click the drop-down menu next to the Formatted Message parameter. From the Webhook dropdown, select Message Content. This will let us view the message received from the Portal webhook in the Automation Log. 
Log Message Content

2. Get the Webhook URL

Save the Automation and give it a name (E.g. “Portal Spatial Analysis Notification”). Once saved, double-click the Webhook trigger to reopen its parameters. Copy the new Webhook URL to your clipboard. Later in the tutorial, this URL will connect our Automation to the ArcGIS Enterprise webhook. 
Webhook URL

Start the Automation. You will receive an error in the ArcGIS Server if you try to create the webhook without first starting the Automation! Leave FME Flow open, as we will come back to it shortly.
Start

3. Create the Webhook

Geoprocessing webhooks are created in the ArcGIS Server Administrator Directory . The URL will be in the format:

https://webadaptorhost.domain.com/webadaptorname/admin

For example, if your ArcGIS Portal web adaptor machine name is gis, domain is safeville.gov, and web adaptor name is arcgis, then the URL would be:  

https://gis.safeville.gov/arcgis/admin 


In a new window or tab in your browser, log in to your ArcGIS Server’s Administrator Directory. In this example, a named user account is used, but you can also use a Portal token.  
Login to ArcGIS Server

Click services from the list of Resources.
Services

Click System from the list of Folders.
System

The System folder contains geoprocessing services that are deployed with ArcGIS Server. Click SpatialAnalysisTools.
Spatial Analysis Tools

Scroll to the bottom of the page and click webhooks from the list of Resources.
Webhooks

Click Create.
Create

Enter the webhook properties:

  • Name: provide a unique, descriptive name for the webhook. If you have multiple FME Flow instances, it’s a good idea to include the FME Flow host name in the webhook name. 
  • Signature Key: leave this empty. The signature key can be used to authenticate webhook messages with the receiver. This is not currently supported by FME Flow.  
  • HookUrl: paste in the Webhook URL from the FME Flow Automation Trigger
  • Content Type: application/json
  • Payload Format: PJSON - this will ‘prettify’ the JSON sent to FME Flow, making it easier to read.
  • Active: True

Webhook Properties

Click Create. The webhook is now active.
Webhook active

Note: if you receive an error about the URL not being reachable, make sure that you started the Automation in step 2.

In FME Flow, view the Automation Log File from within the Automation we just created.
Automation Log

There is a message showing that the ArcGIS Enterprise Portal Webhook has been registered.
Webhook registered

4. Test the webhook

ArcGIS Enterprise Portal will now send a webhook message to FME Flow every time the Spatial Analysis service completes. To test, log into your Portal and open a hosted feature layer in Map Viewer Classic. In this example, a layer of transit station points.

Click Analysis and then Create Buffers. 
Create Buffers

Enter the Buffer parameters and click Run Analysis. 
Buffer Params

Wait for the process to complete then view the FME Flow Automation Log File again (or refresh it, if you still have it open).

The log should now show a new entry with the JSON webhook payload from the Portal showing the task that was run and whether it succeeded or not. 
Automation log message

In the next part, we will create a workspace to parse this JSON and send the information in an email. 

Note: running a hosted geoprocessing service with ArcGIS Pro will also trigger a webhook created for that service. 
 

Part 2: Create a Workspace to Process the Webhook Payload

In this part, we’ll create a workspace in FME Workbench that can be run by the FME Flow Automation when the webhook trigger is fired. The workspace will parse the JSON webhook message and provide the attributes to an FME Flow Automation writer so that they can be included in an email sent from the Automation.

1. Create a New Workspace

Create a new workspace in FME Workbench. Save the workspace as geoprocessingwebhook.fmw. 
 
In the Navigator, right-click User Parameters and select Manage User Parameters. 
Click the Insert button and choose Text.
Add parameter

On the right side of the Parameter Manager, enter the following parameter properties:

  • Parameter Identifier: webhook_message
  • Prompt: Webhook Message
  • Published: Enabled
  • Required: Disabled
  • Disable Attribute Assignment: Disabled
  • Default Value: Paste the webhook JSON payload, from the opening curly brace to the closing one, from the Automation’s log file. This will be used for testing our workspace. 

Click OK.
Configure Param

2. Get the Webhook Response Message
 
Search for the Creator transformer and add it to the canvas. This will start the workspace in the Automation when the webhook is triggered.
 
We can get the webhook message using the ParameterFetcher transformer. Attach a ParameterFetcher transformer to the Creator.
 
Open the ParameterFetcher parameters. For the Parameter Name, open the drop-down menu and select the webhook_message user parameter that was created in the prior step. Name the Target Attribute webhook_message. Click OK. 
Parameter Fetcher

Run the workspace with Feature Caching enabled and inspect the ParameterFetcher output. A new webhook_message attribute has been created with the response text we pasted into the webhook_message’s default value. The webhook can now be passed from FME Flow to our workspace.
View webhook message

3. Parse the JSON

The webhook JSON keys and values need to be turned into features. Double-click the webhook_message value in Visual Preview and format it as JSON; this is done by clicking on the Syntax Highlighting button in the bottom left corner and selecting JSON. 
View JSON

Add a JSONFlattener and connect it to the ParameterFetcher. 
JSONFlattener

Configure the JSONFlatterner:

  • JSON Document: webhook_message
  • Flatten Objects/Arrays: Yes

Configure JSON Flattener

Run the workspace again to build the feature cache. Inspect the JSONFlatterner output in Visual Preview. Select the feature and enable the Feature Information window. The attributes have been created from the JSON; they just need to be exposed. 
View features

Add an AttributeExposer to the canvas and connect it to the JSONFlattener. Expose the following attributes:

  • events{0}.eventType
  • events{0}.when
  • folderName
  • jobId
  • name
  • serviceName
  • serviceType
  • statusURL

Attribute Exposer

4. Convert the Date

ArcGIS Enterprise stores datetimes in an integer format called Epoch (or UNIX). We can convert this format to a readable format in FME using a DateTimeConverter. But, if we try to do this with the date time in the events{0}.when attribute as-is, we’ll get this error:
Date error
 
The datetime returned from ArcGIS Enterprise has milliseconds. 
 
Add an AttributeCreator and connect it to the AttributeExposer.  
Attribute Creator

Open the parameters. Create a new attribute named Formatted_Date. For the Attribute Value, we’ll use an FME Math Function
Click the ellipses (...) to display the Text Editor. 
Scroll down the function list and find the Math Functions. 
From the list of Math Functions, double-click div. Enter the formula below to divide the date integer by 1000 and convert the milliseconds to seconds. 

@div(@Value(events{0}.when),1000)

Math function

Click OK on the Text Editor and OK on the AttributeCreator. 

Now the date can be converted by FME. Add a DateTimeConverter and connect it to the AttributeCreator. 
 
Enter the following parameters in the DateTimeConverter:

  • Datetime Attributes: formattedDate
  • Input Format: %s$ (Epoch Time)
  • Output Format: ISO 

Configure date time converter

Please note: the ISO Output Format displays a Z at the end of the time, for Zulu. This is because the date is in UTC time, which is synonymous with Zulu time. For the scope of this article, we won’t format the date any further, but please read Handling Local Timezones, UTC, Daylight Savings Time, and Leap Units if you’d like to learn more.
 
Run the workspace and inspect the DateTimeConverter output. There is now a formattedDate we can include in our email.
Formatted Date

5. Get Additional Info About the Geoprocessing Service Job

You may have noticed that the webhook message contains a statusURL. This URL can be called to get additional information about the geoprocessing service job. 

To make getting this information more simple, we can use the ArcGISPortalWebhookDataGetter from the FME Hub. This transformer will return a feature with any messages from the geoprocessing service job. This is especially useful for errors or if you have a custom geoprocessing service that returns custom messages.

Add the ArcGISPortalWebHookDataGetter to the canvas and connect it to the DateTimeConverter’s output.
ArcGIS Portal Webhook Data Getter

Configure the parameters:

Configure Webhook Data Getter

This custom transformer is embedded by default, which makes it more simple to publish the workspace to FME Flow later. To learn more about working with custom transformers in FME Flow, please read Custom Transformers and FME Flow

Run the workspace and inspect the output from the GeoprocessingMessages port. The esriGeoprocessingJobMessages attribute will list any additional messages. Since the buffer applied in this example was successful, there are none. 
View ArcGIS Portal Webhook Output

Below is an example of geoprocessing job messages for a failed job.
Failed geoprocessing job messages

6. Clean up Attributes

Add an AttributeManager to the canvas and connect it to the GeoprocessingMessages output port of the ArcGISPortalWebhookDataGetter. Clean up the schema so that only fields we want to include in the notification remain. Remove:

  • _creation_instance
  • webhook_message
  • events{0}.eventType
  • events{0}.when
  • events{}.eventType
  • events{}.when
  • folderName
  • statusURL

Attribute Manager

7. Write to FME Flow Automation

Add an FME Flow Automations writer to the canvas. The schema has already been cleaned up, so the Feature Type Definition can be left as Automatic. Click OK.
Automation Writer

In the Feature Type dialog, name the Feature Type SpatialAnalysisTools_Job to reflect the geoprocessing service that the ArcGIS Enterprise Portal webhook was created for. Click OK.
Automation writer feature type name

Connect the Automations writer to the AttributeManager. 
Canvas with automation writer

Run the workspace again and make sure you receive no errors. 

8. Publish to FME Flow

Now that our workspace is complete, we need to publish it to FME Flow so we can use it in an Automation. If this is your first time doing this, please read our article on publishing workspaces to FME Flow before proceeding. It is a good idea to save your workspace before publishing. 
 
In FME Workbench, click Publish.
 Publish button

Connect to your FME Flow instance (if you are not already connected) and follow the publish wizard. Choose a repository to upload the workspace to (or create a new repository). On the final step before publishing, register the Job Submitter service only.
Register services
 
Click Publish and check the Translation Log to make sure everything was published to FME Flow successfully.
 

Part 3: Complete the FME Flow Automation

All that is left to do is complete the Automation so we can start sending notifications.

1. Confirm that ArcGIS Enterprise Portal Web Connection has been Authorized

Since the workspace connects to ArcGIS Enterprise Portal to retrieve additional job details, ensure your Portal connection is working. 

In FME Flow, click Web Connections under Connections & Parameters. From the list of web connections, click the name of your ArcGIS Enterprise Portal web connection.
Web connections

Click Authorize and allow the request for permission from Esri when prompted. Once authorized, you should see a green check mark beside the connection name.
Authorize

If the Authorize button is inactive or you receive an error, please ensure you have configured your ArcGIS Enterprise Portal web service correctly in both FME Workbench and FME Flow. Please see: How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0)

2. Create a Workspace Action

Open the Automation we previously created. Stop the Automation if it is still running.

Leave the Log action in place. It will be useful to continue logging the webhook payload if there is an issue. Add a new Action to the canvas and connect the Webhook Trigger’s Action Succeeded port to it. 
Automation canvas

Open the Action and choose Run a Workspace. Choose the repository where your workspace was published. You will notice that the Webhook Message field is already filled in. This is the test payload that we used in the parameter default value, but we want the message content instead. Open the drop-down menu for Webhook Message, expand Webhook, and select Message Content. Click Apply.
Configure run workspace

After adding the action, you will see a port named SpatialAnalysisTools_Job on the Run Workspace Action. This port is available because the workspace we created uses the FME Flow Automations Writer, which lets a workspace pass attributes to an Automation. For more information, see Routing Data Between Workspaces in Automations
Automation writer output

3. Create an External Action to Send the Email

Connect an external action to the SpatialAnalysisTools_Job port of the FME Workspace action.
Add external email action


For the Action, choose Email (send). The example below uses Gmail to send the email. Click Load Template and choose Gmail. Enter the SMTP Account and Password. In the Email To field, enter the email(s) to notify when a Portal item is shared with everyone (use your own email for testing).
Configure external email action

Create an Email Subject using the Text Editor to access the attributes from the workspace.
Create email subject

In the Text Editor, click an attribute name to add it to the subject. You can use this approach to create a formatted subject with key info. For example:

[ArcGIS Portal] Geoprocessing service {route.SpatialAnalysisTools_Job.serviceName} job {route.SpatialAnalysisTools_Job.jobId} has ended with status {route.SpatialAnalysisTools_Job.esriGeoprocessingJobStatus}

Email subject

You can take the same approach to create a formatted Email Body. For example: 

Geoprocessing job {route.SpatialAnalysisTools_Job.jobId} details:

Status: {route.SpatialAnalysisTools_Job.esriGeoprocessingJobStatus}
Completed Date (UTC): {route.SpatialAnalysisTools_Job.formattedDate}
Task: {route.SpatialAnalysisTools_Job.taskName}
Service Name: {route.SpatialAnalysisTools_Job.serviceName}
Service Type: {route.SpatialAnalysisTools_Job.serviceType}

Messages:
{route.SpatialAnalysisTools_Job.esriGeoprocessingJobMessages}


==============================
Triggered by webhook: {route.SpatialAnalysisTools_Job.name}

Email body

Click Apply to save the External Email Action parameters. Restart the Automation.
Automation started

4. Test the Completed Automation

Let’s test it out. In ArcGIS Enterprise Portal, run another task against the SpatialAnalysisTools geoprocessing service. In this example, the transit station buffers created earlier in Part 1 are dissolved. 
Dissolve buffers

Wait for the geoprocessing service job to complete. The FME Flow Automation sends an email notifying of job completion.
Email notification

Note: if you don’t receive an email, check the Automation Log for errors.

Your ArcGIS Enterprise Portal webhook Automation is complete! With this setup, you will automatically be sending an email notification every time the SpatialAnalysisTools geoprocessing service completes. You can use the same approach to setup webhook Automations to monitor a custom geoprocessing services hosted on your ArcGIS Enterprise Portal.
 

Additional Resources

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.