Automate ArcGIS Enterprise Portal Administration with Organization Webhooks and FME Flow

FME Version

Introduction

Esri’s ArcGIS Portal is a component of ArcGIS Enterprise that allows you to host your organization’s maps and data on the web. Anyone in the organization with the right permissions and a web browser can view and manage spatial data, which makes sharing spatial data easy.

ArcGIS Portal is often just one of many applications in an organization’s ecosystem. The responsibility of managing all of its datasets, users, groups, and items often falls on an administrator. FME offers a solution for automating this process.

FME Flow Automations run automated workflows in response to different kinds of events, including incoming messages. Organization Webhooks can be used to monitor item, group, and user activity in ArcGIS Portal and send out messages about that activity. Connecting organization webhooks to automations allows us to build workflows that respond to activity and events occurring in ArcGIS Portal in near real-time.

For example, when a Feature Layer is uploaded to Portal, we may want to initiate the following workflows:

Send out a notification to stakeholders
Synchronize with another database
Validate and clean-up datasets

In the following tutorial, we will set up an FME Flow Automation to receive a message every time an item in ArcGIS Portal is shared externally. However, organization webhooks supports dozens of other different trigger events. If you’re interested in using FME to monitor other ArcGIS Enterprise portal items, groups, and users, you can implement those using a similar approach to this example.

Requirements

An ArcGIS Enterprise Portal Web Connection
An ArcGIS Portal account with the Organization webhooks privilege or assigned the default administrator role .
FME Flow and ArcGIS Enterprise Portal must be able to communicate within the same network OR be configured for public access. For more information, refer to your organization’s IT and the Enabling FME Flow for Public Access article.
FME Flow must have HTTPS enabled (the Portal webhook payload is delivered via HTTPS)
ArcGIS Enterprise Portal must trust the certificate used by your FME Flow

Step-by-Step Instructions

In this scenario, an ArcGIS admin works for an organization that provides datasets to the general public. As part of their responsibilities, the admin must ensure all employees are publishing datasets that adhere to governance and accessibility guidelines. But as the organization grows, manually reviewing each shared dataset has become too time-consuming.

After learning that FME can be used to receive real-time messages about ArcGIS Portal item activity, the admin is interested in building an automated solution with FME. This workflow will send out notifications the moment each item is shared, so that it can be reviewed and validated.

This tutorial will provide instructions for creating the organization webhook in ArcGIS Portal, building an FME Flow Automation to receive webhook messages, and creating a workspace that processes the webhook messages.

Part 1: Create the ArcGIS Enterprise Portal Organization Webhook

In Part 1, we build an automation in FME Flow that receives webhook messages from ArcGIS Portal. In ArcGIS Portal, we create the organization webhook that will send a message every time a feature layer is shared externally.

1. Configure the Automation in FME Flow
Start by building a new Automation in FME Flow that will receive messages from the ArcGIS Enterprise Portal.

Set the Trigger to Webhook (triggered) and click Apply. Attach a Log a Message Action to the Success port (the checkmark) of the Trigger.

Click the drop-down menu next to the Formatted Message parameter. From the Webhook dropdown, select Message Content. This will let us view the message received from the Portal webhook in our Automation Log.

2. Get the Webhook URL
Save the Automation and give it a name (E.g. “Portal External Sharing”). Once saved, double-click the Webhook trigger to reopen its Parameters. Copy the new Webhook URL to your clipboard. Later in the tutorial, this URL will connect our Automation to the ArcGIS Enterprise Portal webhook.

Start the Automation. You will receive an error in the ArcGIS Enterprise Portal if you try to create the webhook without first starting the Automation! Leave FME Flow open, as we will come back to it shortly.

3. Create the Webhook
In a new window or tab in your browser, login to your ArcGIS Enterprise Portal’s REST API Directory as an administrator. The URL will be in the format:

https://webadaptorhost.domain.com/webadaptorname/sharing/rest

For example, if your ArcGIS Portal web adaptor machine name is gis, domain is safeville.gov, and web adaptor name is portal, then the URL would be:

https://gis.safeville.gov/portal/sharing/rest

After logging in, you will see the admin home page. If you do not, click admin in the top right.

Scroll down on the Organization ID page. From the list of Child Resources, click Webhooks.

Scroll down on the Organization ID page. From the list of Child Resources, click Webhooks.

Click Create Webhook.

All ArcGIS Enterprise Portal webhooks for the organization are created on this page.

Paste the Webhook URL from the clipboard (copied in step 2) into the Payload URL.
Enter a unique name for the webhook.
Under Which events would you like to trigger this webhook?, select “Let me define manually”.
Enter the ArcGIS Portal Trigger Event for which you want to send a message to FME Flow. In this example, we’ll enter ”/items/share” in order to receive a message whenever any item in our organization is shared.

Click Create Webhook. The webhook is now active.

Notes:

If you get the error “Unable to create webhook for the account due to 'null’” when creating the webhook, your login token likely expired. Simply click login in the top right corner and re-login as an administrator.
If you get an error related to SSL configuration, e.g. ‘Exception: PKIX path building failed: host unable to find valid certification path to requested target’, please ensure that your FME Flow SSL certificate has been imported into your ArcGIS Portal following the instructions from Esri .

In FME Flow, view the Automation Log File from within the automation we just created.

There is a message showing that the ArcGIS Enterprise Portal Webhook has been registered.

4. Test the webhook
Share an item in your ArcGIS Enterprise Portal

Note: to share an editable feature layer with Everyone (public), you must first enable Public Data Collection in the feature layer’s settings

Check the Automation Log File again in FME Flow (you may need to refresh the page).
FME Flow has received a webhook message (in JSON) letting us know which item has been shared and who shared it. The message only contains the ID for the item though; in the next parts, we will create a workspace that gets the item name and uses it in a notification.

Part 2: Create a Workspace to Process the Webhook Payload

In Part 2, we’ll create a workspace in FME Workbench that can be run by the FME Flow Automation every time the webhook trigger receives a message from ArcGIS Portal. The workspace will get the item details via an ArcGIS REST API call and write them to an FME Flow Automation writer so that they can be included in an email.

1. Create a New Workspace and User Parameters
Open a blank workspace in FME Workbench. Save the workspace as portalwebhook.fmw.

In the Navigator, right-click User Parameters and select Manage User Parameters. Click the Insert button and choose Text.

On the right side of the Parameter Manager, enter the following parameter properties:

Parameter Identifier: webhook_message
Prompt: Webhook Message
Published: Enabled
Required: Disabled
Disable Attribute Assignment: Disabled
Default Value: Paste the webhook JSON payload, from the opening curly brace to the closing one, from the Automation’s log file (see Part 1, Step 4). This will be used for testing our workspace.

Click OK.

2. Get the Webhook Response Message
Search for the Creator transformer and add it to the canvas. This will start the workspace in the Automation when the webhook is triggered.

We can create a feature from the webhook message using the ParameterFetcher transformer. Attach a ParameterFetcher transformer to the Creator.
Open the ParameterFetcher parameters.
For the Parameter Name, open the drop-down menu and select the “webhook_message” user parameter that was created in the prior step.
Name the Target Attribute “_webhook_message”.
Click OK.

Run the workspace with Feature Caching Enabled and inspect the ParameterFetcher output in the Visual Preview window. A new webhook_message attribute has been created with the response text we pasted into the webhook_message’s default value. Now, the FME Flow Automation Webhook Trigger can pass the ArcGIS Portal message into our workspace for processing.

3. Parse the JSON
The webook’s JSON keys and values need to be parsed into attributes. To view the webhook message as JSON, double-click the _webhook_message value in Visual Preview, click on the Syntax Highlighting button in the bottom left corner and select JSON.
For this example, the only JSON keys we need are in the “events” array: “id”, “operation” and “sharedToGroups” (under “properties”).

Add a JSONFragmenter and connect it to the ParameterFetcher.

Configure the JSONFragmenter:

JSON Attribute: webhook_message
JSON Query: json[“events”][*]
Flatten Query Result into Attributes: Yes
Recursively Flatten Objects/Arrays: Yes

In the “Attributes to Expose” parameter, enter the following keys on new lines:

id
operation
properties.sharedToGroups{}
source
username

Select OK to close the transformer.

Run the workspace again to build the feature cache.

Inspect the JSONFragmenter output in Visual Preview. Select the feature and enable the Feature Information window to see the attributes created from the JSON.

4. Check if the Item Has Been Shared with Everyone
In this example, we only want to send a notification if an item has been shared with Everyone. Before letting the workflow continue, let’s check that condition. Connect a ListSearcher to the JSONFragmenter. Search the list attribute properties.sharedToGroup for the string ‘Everyone’.

Run the workspace. A feature is output from the Found port. If a feature is not found, we won’t continue the transformation.

Next, flatten the sharedToGroups list into a single attribute, to make it easier to work with and display to the user.

Connect a ListConcatenator to the ListSearcher Found port. For the List Attribute, choose the properties.sharedToGroups{} list and for the Destination attributes, enter sharedToGroups.

The new attribute can be seen in the Visual Preview when the workspace is run.

5. Create the ArcGIS Enterprise Portal Web Connections
The ArcGIS REST API URL that we will call to get more information about the shared item (provided in the webhook message) requires the item id.

An ArcGIS Portal Web Connection is required to connect to ArcGIS Portal. If you do not already have an ArcGIS Portal Web Connection, please read How to Create an FME Web Connection for ESRI ArcGIS Portal Feature Service for instructions.

Add an HTTPCaller to the canvas and connect it to the ListConcantenator.

Instead of hardcoding the value, the request URL will reference the item id attribute value since it will be different each time. For the Request URL, give it the value of https://yourportaldomain.com/portal/sharing/rest/content/items/@Value(id) where “yourportaldomain.com” is the domain for your Portal.

Configure the rest of the HTTPCaller as follows:

HTTP Method: GET
Query String Parameters:
- Name: f
- Value: json (this will return the data in JSON format)
Response Body Attribute: _response_body
Use Authentication: Yes
- Authentication Method: Web Connection
- Web Connection: the ArcGIS Enterprise Portal Web Connection you setup for FME Workbench

Run the workspace again and inspect the HTTPCaller output in Visual Preview. Preview the _response_body attribute; it should contain details about the item that was shared, such as the name and description.

6. Parse JSON and Cleanup Attributes
Connect a JSONFlattener to the HTTPCaller output. Set the JSON Document to the _response_body attribute returned from the HTTPCaller and set Recursively Flatten Objects/Arrays to Yes. Expose the following attributes from the JSON: title, description and type.

To keep the output clean, connect an AttributeKeeper to the JSONFlattener and only keep the following attributes:

description
id
operation
sharedToGroups
title
type
username

7. Write to FME Flow Automation and Upload to FME Flow
Add an FME Flow Automation writer to the canvas. The Feature Type Definition can be set to Automatic… .

In the Feature Type dialog, name the Feature Type SharedItem. Then click OK.

Connect the AttributeKeeper to the SharedItem Automations writer.

Run the workspace again and make sure you receive no errors.

Bookmarks can be used to clean up the workspace. Add a bookmark around all the transformers from the ParameterFetcher to the ListConcaternator and name it “Process Webhook”. Add another bookmark around the transformers from the HTTPCaller to the AttributeKeeper and name it “Get Item Information”.

The workspace should now look something like this:

8. Publish to FME Flow
Save the workspace and click Publish to open the Publishing Wizard to connect and publish the workspace to FME Flow.

Select your FME Flow connection and choose a repository to upload the workspace to (or create a new repository). Click Next.

In the next window, select your ArcGIS Enterprise Portal Web Connection if it has not already been published to FME Flow. Click Next.

In Register Services, select Job Submitter and click Publish.

Check the Translation Log to make sure everything was published to FME Flow successfully.

Part 3: Complete the FME Server Automation

In Part 3, we finish building the FME Flow Automation so that it runs our workspace and sends an email notification each time it receives a webhook message from ArcGIS Portal.

Firstly, an ArcGIS Enterprise Portal web connection needs to be authorized and working on FME Flow. If the ArcGIS Portal web connection was uploaded for the first time with the workspace, finish the process for authorization on FME Flow.

1. Create a Workspace Action
Open the Automation we created in Part 1. Stop the Automation if it is still running.

Add a new Action to the canvas and connect it to the Webhook Trigger’s Action Success port. Keep the log action attached to the success port so we can see the webhook response in case the workspace fails.

Open the Action and choose Run a Workspace. Choose the repository where your workspace was published and select the workspace.

Once the workspace is selected, you will notice that the Webhook Message parameter is already filled in with the test payload used as our workspace’s user parameter default value. Instead, we want to run the automated workspace with real-time message content. Open the drop-down menu for Webhook Message, expand Webhook Keys, and select “Message Content”. Click Apply.

After adding the workspace, you will see a port named SharedItem on the Run Workspace Action. This port is available because the workspace we created writes to an FME Flow Automation Writer, which creates a port from the Feature Type name. For more information about the FME Flow Automation Writer, see the Routing Data Between Workspaces in Automations article.

2. Create an External Action to Send the Email Notification
Connect an external action to the SharedItem port of the FME Workspace action. For the Action, choose Email (send).

The example below uses Gmail to send the email. Click Load Template and choose Gmail. Enter the SMTP Account and Password. In the Email To field, enter the email(s) to notify when a Portal item is shared with everyone (use your own email for testing).

Create an Email Subject using the Text Editor to access the attributes from the workspace.

In the Text Editor, click an attribute name to add it to the subject.

You can take the same approach to create a formatted Email Body.
The following item from the ArcGIS Enterprise Portal has been shared with {route.SharedItem.sharedToGroups}

Title: {route.SharedItem.title}
Description: {route.SharedItem.description}
Type: {route.SharedItem.type}
ID: {route.SharedItem.id}

Shared By: {route.SharedItem.username}

Click Apply to save the External Action parameters. Save and Start the Automation.

3. Test the Portal External Sharing Automation
Share any item from your ArcGIS Enterprise portal. Give a few seconds and check your email.

Note: if you don’t receive an email, check the Automation Log for errors.

Your ArcGIS Portal organization webhook Automation is complete!

In this workflow, we built an FME Flow Automation to send an email whenever an item in your Portal is shared with everyone. The ArcGIS admin is able to remain vigilant about what gets shared with the public, while saving valuable time out of their day.

Using ArcGIS Enterprise Portal webhooks with the ArcGIS Portal Reader/Writer, ArcGISOnlineConnector, and the HTTPCaller (for ArcGIS REST API requests) you can automate many other administrative tasks.