FME Version
Introduction
Esri’s ArcGIS Server, which is included with ArcGIS Enterprise, offers webhooks that can send data to FME Flow when specific events occur in near real-time. As of ArcGIS Enterprise 11.1, Esri has introduced geoprocessing webhooks.
A geoprocessing webhook will send information about a geoprocessing service when it completes on ArcGIS Server. For example, we may want FME Flow to start a workflow when a long-running geoprocess completes on ArcGIS Server. This may include:
- Sending out a notification on geoprocessing success or failure
- Initiating a process in another application
- Initiating other downstream processes
Unlike organization or feature service webhooks, geoprocessing webhooks do not have specific event triggers that can be defined. The webhook will send data every time the subscribed geoprocessing service completes.
In the following tutorial, we will set up an FME Flow Automation to send an email every time the SpatialAnalysisTools geoprocessing service completes. This is a system service that comes with ArcGIS Server and enables basic spatial tasks , like creating buffers. This is a simple example for the purpose of this article. In reality, you may wish to monitor a custom geoprocessing service created from a model in ArcGIS Pro, for example.
Requirements
- ArcGIS Enterprise 11.1 or greater, which includes ArcGIS Server and Portal
- Must be using FME 2022.2.7 or 2023.0.0.3 since older versions do not support web connections to ArcGIS Enterprise 11.1. See Known Issue - FME fails to create a portal web connection to ArcGIS Enterprise 11.1
- Access to your organization’s ArcGIS Enterprise Server Administrator Directory with the default administrator role or a custom role that has both the ‘geoprocessing’ and ‘publish server-based layers’ privileges
- FME Flow and ArcGIS Server must be on the same network to communicate
- FME Flow must have HTTPS enabled (ArcGIS Server delivers the webhook payload via HTTPS)
- ArcGIS Enterprise Server must also have HTTPS enabled
- ArcGIS Enterprise Server must trust the SSL certificate used by your FME Flow instance. If it does not, the certificate must be imported.
Step-by-Step Instructions
Part 1: Create the ArcGIS Enterprise Portal Webhook
1. Configure the Automation in FME Flow
Start by building a new Automation in FME Flow that will receive messages from the ArcGIS Enterprise Portal.
Set the Trigger to Webhook (triggered), leave Required Authentication unchecked, and click Apply. Attach a Log a Message Action to the Action Succeeded port (the checkmark) of the Trigger.
Click the drop-down menu next to the Formatted Message parameter. From the Webhook dropdown, select Message Content. This will let us view the message received from the Portal webhook in the Automation Log.
2. Get the Webhook URL
Save the Automation and give it a name (E.g. “Portal Spatial Analysis Notification”). Once saved, double-click the Webhook trigger to reopen its parameters. Copy the new Webhook URL to your clipboard. Later in the tutorial, this URL will connect our Automation to the ArcGIS Enterprise webhook.
Start the Automation. You will receive an error in the ArcGIS Server if you try to create the webhook without first starting the Automation! Leave FME Flow open, as we will come back to it shortly.
3. Create the Webhook
Geoprocessing webhooks are created in the ArcGIS Server Administrator Directory . The URL will be in the format:
https://webadaptorhost.domain.com/webadaptorname/admin
For example, if your ArcGIS Portal web adaptor machine name is gis, domain is safeville.gov, and web adaptor name is arcgis, then the URL would be:
https://gis.safeville.gov/arcgis/admin
In a new window or tab in your browser, log in to your ArcGIS Server’s Administrator Directory. In this example, a named user account is used, but you can also use a Portal token.
Click services from the list of Resources.
Click System from the list of Folders.
The System folder contains geoprocessing services that are deployed with ArcGIS Server. Click SpatialAnalysisTools.
Scroll to the bottom of the page and click webhooks from the list of Resources.
Click Create.
Enter the webhook properties:
- Name: provide a unique, descriptive name for the webhook. If you have multiple FME Flow instances, it’s a good idea to include the FME Flow host name in the webhook name.
- Signature Key: leave this empty. The signature key can be used to authenticate webhook messages with the receiver. This is not currently supported by FME Flow.
- HookUrl: paste in the Webhook URL from the FME Flow Automation Trigger
- Content Type: application/json
- Payload Format: PJSON - this will ‘prettify’ the JSON sent to FME Flow, making it easier to read.
- Active: True
Click Create. The webhook is now active.
Note: if you receive an error about the URL not being reachable, make sure that you started the Automation in step 2.
In FME Flow, view the Automation Log File from within the Automation we just created.
There is a message showing that the ArcGIS Enterprise Portal Webhook has been registered.
4. Test the webhook
ArcGIS Enterprise Portal will now send a webhook message to FME Flow every time the Spatial Analysis service completes. To test, log into your Portal and open a hosted feature layer in Map Viewer Classic. In this example, a layer of transit station points.
Click Analysis and then Create Buffers.
Enter the Buffer parameters and click Run Analysis.
Wait for the process to complete then view the FME Flow Automation Log File again (or refresh it, if you still have it open).
The log should now show a new entry with the JSON webhook payload from the Portal showing the task that was run and whether it succeeded or not.
In the next part, we will create a workspace to parse this JSON and send the information in an email.
Note: running a hosted geoprocessing service with ArcGIS Pro will also trigger a webhook created for that service.
Part 2: Create a Workspace to Process the Webhook Payload
In this part, we’ll create a workspace in FME Workbench that can be run by the FME Flow Automation when the webhook trigger is fired. The workspace will parse the JSON webhook message and provide the attributes to an FME Flow Automation writer so that they can be included in an email sent from the Automation.
1. Create a New Workspace
Create a new workspace in FME Workbench. Save the workspace as geoprocessingwebhook.fmw.
In the Navigator, right-click User Parameters and select Manage User Parameters.
Click the Insert button and choose Text.
On the right side of the Parameter Manager, enter the following parameter properties:
- Parameter Identifier: webhook_message
- Prompt: Webhook Message
- Published: Enabled
- Required: Disabled
- Disable Attribute Assignment: Disabled
- Default Value: Paste the webhook JSON payload, from the opening curly brace to the closing one, from the Automation’s log file. This will be used for testing our workspace.
Click OK.
2. Get the Webhook Response Message
Search for the Creator transformer and add it to the canvas. This will start the workspace in the Automation when the webhook is triggered.
We can get the webhook message using the ParameterFetcher transformer. Attach a ParameterFetcher transformer to the Creator.
Open the ParameterFetcher parameters. For the Parameter Name, open the drop-down menu and select the webhook_message user parameter that was created in the prior step. Name the Target Attribute webhook_message. Click OK.
Run the workspace with Feature Caching enabled and inspect the ParameterFetcher output. A new webhook_message attribute has been created with the response text we pasted into the webhook_message’s default value. The webhook can now be passed from FME Flow to our workspace.
3. Parse the JSON
The webhook JSON keys and values need to be turned into features. Double-click the webhook_message value in Visual Preview and format it as JSON; this is done by clicking on the Syntax Highlighting button in the bottom left corner and selecting JSON.
Add a JSONFlattener and connect it to the ParameterFetcher.
Configure the JSONFlatterner:
- JSON Document: webhook_message
- Flatten Objects/Arrays: Yes
Run the workspace again to build the feature cache. Inspect the JSONFlatterner output in Visual Preview. Select the feature and enable the Feature Information window. The attributes have been created from the JSON; they just need to be exposed.
Add an AttributeExposer to the canvas and connect it to the JSONFlattener. Expose the following attributes:
- events{0}.eventType
- events{0}.when
- folderName
- jobId
- name
- serviceName
- serviceType
- statusURL
4. Convert the Date
ArcGIS Enterprise stores datetimes in an integer format called Epoch (or UNIX). We can convert this format to a readable format in FME using a DateTimeConverter. But, if we try to do this with the date time in the events{0}.when attribute as-is, we’ll get this error:
The datetime returned from ArcGIS Enterprise has milliseconds.
Add an AttributeCreator and connect it to the AttributeExposer.
Open the parameters. Create a new attribute named Formatted_Date. For the Attribute Value, we’ll use an FME Math Function:
Click the ellipses (...) to display the Text Editor.
Scroll down the function list and find the Math Functions.
From the list of Math Functions, double-click div. Enter the formula below to divide the date integer by 1000 and convert the milliseconds to seconds.
@div(@Value(events{0}.when),1000)
Click OK on the Text Editor and OK on the AttributeCreator.
Now the date can be converted by FME. Add a DateTimeConverter and connect it to the AttributeCreator.
Enter the following parameters in the DateTimeConverter:
- Datetime Attributes: formattedDate
- Input Format: %s$ (Epoch Time)
- Output Format: ISO
Please note: the ISO Output Format displays a Z at the end of the time, for Zulu. This is because the date is in UTC time, which is synonymous with Zulu time. For the scope of this article, we won’t format the date any further, but please read Handling Local Timezones, UTC, Daylight Savings Time, and Leap Units if you’d like to learn more.
Run the workspace and inspect the DateTimeConverter output. There is now a formattedDate we can include in our email.
5. Get Additional Info About the Geoprocessing Service Job
You may have noticed that the webhook message contains a statusURL. This URL can be called to get additional information about the geoprocessing service job.
To make getting this information more simple, we can use the ArcGISPortalWebhookDataGetter from the FME Hub. This transformer will return a feature with any messages from the geoprocessing service job. This is especially useful for errors or if you have a custom geoprocessing service that returns custom messages.
Add the ArcGISPortalWebHookDataGetter to the canvas and connect it to the DateTimeConverter’s output.
Configure the parameters:
- Webhook Response Message: webhook_message
- Select a Web Connection: your ArcGIS Enterprise Portal web connection. If you have not created a connection, please see How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0)
This custom transformer is embedded by default, which makes it more simple to publish the workspace to FME Flow later. To learn more about working with custom transformers in FME Flow, please read Custom Transformers and FME Flow .
Run the workspace and inspect the output from the GeoprocessingMessages port. The esriGeoprocessingJobMessages attribute will list any additional messages. Since the buffer applied in this example was successful, there are none.
Below is an example of geoprocessing job messages for a failed job.
6. Clean up Attributes
Add an AttributeManager to the canvas and connect it to the GeoprocessingMessages output port of the ArcGISPortalWebhookDataGetter. Clean up the schema so that only fields we want to include in the notification remain. Remove:
- _creation_instance
- webhook_message
- events{0}.eventType
- events{0}.when
- events{}.eventType
- events{}.when
- folderName
- statusURL
7. Write to FME Flow Automation
Add an FME Flow Automations writer to the canvas. The schema has already been cleaned up, so the Feature Type Definition can be left as Automatic. Click OK.
In the Feature Type dialog, name the Feature Type SpatialAnalysisTools_Job to reflect the geoprocessing service that the ArcGIS Enterprise Portal webhook was created for. Click OK.
Connect the Automations writer to the AttributeManager.
Run the workspace again and make sure you receive no errors.
8. Publish to FME Flow
Now that our workspace is complete, we need to publish it to FME Flow so we can use it in an Automation. If this is your first time doing this, please read our article on publishing workspaces to FME Flow before proceeding. It is a good idea to save your workspace before publishing.
In FME Workbench, click Publish.
Connect to your FME Flow instance (if you are not already connected) and follow the publish wizard. Choose a repository to upload the workspace to (or create a new repository). On the final step before publishing, register the Job Submitter service only.
Click Publish and check the Translation Log to make sure everything was published to FME Flow successfully.
Part 3: Complete the FME Flow Automation
All that is left to do is complete the Automation so we can start sending notifications.
1. Confirm that ArcGIS Enterprise Portal Web Connection has been Authorized
Since the workspace connects to ArcGIS Enterprise Portal to retrieve additional job details, ensure your Portal connection is working.
In FME Flow, click Web Connections under Connections & Parameters. From the list of web connections, click the name of your ArcGIS Enterprise Portal web connection.
Click Authorize and allow the request for permission from Esri when prompted. Once authorized, you should see a green check mark beside the connection name.
If the Authorize button is inactive or you receive an error, please ensure you have configured your ArcGIS Enterprise Portal web service correctly in both FME Workbench and FME Flow. Please see: How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0)
2. Create a Workspace Action
Open the Automation we previously created. Stop the Automation if it is still running.
Leave the Log action in place. It will be useful to continue logging the webhook payload if there is an issue. Add a new Action to the canvas and connect the Webhook Trigger’s Action Succeeded port to it.
Open the Action and choose Run a Workspace. Choose the repository where your workspace was published. You will notice that the Webhook Message field is already filled in. This is the test payload that we used in the parameter default value, but we want the message content instead. Open the drop-down menu for Webhook Message, expand Webhook, and select Message Content. Click Apply.
After adding the action, you will see a port named SpatialAnalysisTools_Job on the Run Workspace Action. This port is available because the workspace we created uses the FME Flow Automations Writer, which lets a workspace pass attributes to an Automation. For more information, see Routing Data Between Workspaces in Automations .
3. Create an External Action to Send the Email
Connect an external action to the SpatialAnalysisTools_Job port of the FME Workspace action.
For the Action, choose Email (send). The example below uses Gmail to send the email. Click Load Template and choose Gmail. Enter the SMTP Account and Password. In the Email To field, enter the email(s) to notify when a Portal item is shared with everyone (use your own email for testing).
Create an Email Subject using the Text Editor to access the attributes from the workspace.
In the Text Editor, click an attribute name to add it to the subject. You can use this approach to create a formatted subject with key info. For example:
[ArcGIS Portal] Geoprocessing service {route.SpatialAnalysisTools_Job.serviceName} job {route.SpatialAnalysisTools_Job.jobId} has ended with status {route.SpatialAnalysisTools_Job.esriGeoprocessingJobStatus}
You can take the same approach to create a formatted Email Body. For example:
Geoprocessing job {route.SpatialAnalysisTools_Job.jobId} details:
Status: {route.SpatialAnalysisTools_Job.esriGeoprocessingJobStatus}
Completed Date (UTC): {route.SpatialAnalysisTools_Job.formattedDate}
Task: {route.SpatialAnalysisTools_Job.taskName}
Service Name: {route.SpatialAnalysisTools_Job.serviceName}
Service Type: {route.SpatialAnalysisTools_Job.serviceType}
Messages:
{route.SpatialAnalysisTools_Job.esriGeoprocessingJobMessages}
==============================
Triggered by webhook: {route.SpatialAnalysisTools_Job.name}
Click Apply to save the External Email Action parameters. Restart the Automation.
4. Test the Completed Automation
Let’s test it out. In ArcGIS Enterprise Portal, run another task against the SpatialAnalysisTools geoprocessing service. In this example, the transit station buffers created earlier in Part 1 are dissolved.
Wait for the geoprocessing service job to complete. The FME Flow Automation sends an email notifying of job completion.
Note: if you don’t receive an email, check the Automation Log for errors.
Your ArcGIS Enterprise Portal webhook Automation is complete! With this setup, you will automatically be sending an email notification every time the SpatialAnalysisTools geoprocessing service completes. You can use the same approach to setup webhook Automations to monitor a custom geoprocessing services hosted on your ArcGIS Enterprise Portal.
Comments
0 comments
Please sign in to leave a comment.