Integrating ArcGIS Enterprise Feature Service Webhooks With FME

Introduction

Esri’s ArcGIS Server, which is included with ArcGIS Enterprise, offers webhooks that can send data to FME Flow when specific events occur in near real-time. As of ArcGIS Enterprise 11.1, Esri has introduced feature service webhooks.

Feature service webhooks allow you to subscribe to different events occurring on a feature service. This includes adds, deletes and updates, making it exceptionally useful for updating downstream systems or tracking changes made to a feature service without needing to constantly poll it.

In this article, we will explain how to create feature service webhooks in ArcGIS Enterprise and integrate them with a simple FME Flow workflow which will notify a user when changes have been made to a feature service. 

To follow along with the instructions in this article, download the shapefile zip file attached to this article and upload it to your own ArcGIS Portal, where you will be creating the feature service webhooks. Alternatively, you can follow along with the instructions using your own feature service.

Requirements

Specific requirements must be met in order for you to use ArcGIS Enterprise Feature Service Webhooks on FME Flow. 

• Must be using ArcGIS Enterprise 11.1 or later for Hosted Feature Service Webhooks
• Must be using ArcGIS Enterprise 11.3 or later for Referenced Feature Service Webhooks
• Must be using FME 2022.2.7 or 2023.0.0.3 since older versions do not support web connections to ArcGIS Enterprise 11.1. See Known Issue - FME fails to create a portal web connection to ArcGIS Enterprise 11.1
• Webhooks can only be created and managed by users assigned the Feature layer privilege or the default administrator role.
• The feature service must include the Editing capability and have change tracking enabled without sync.
• FME Flow and ArcGIS Server must be on the same network to communicate
• FME Flow must have HTTPS enabled (ArcGIS Server delivers the webhook payload via HTTPS)
• ArcGIS Enterprise Server must also have HTTPS enabled
• ArcGIS Enterprise Server must trust the SSL certificate used by your FME Flow instance. If it does not, the certificate must be imported.
• For requirements from Esri, please reference the ArcGIS Webhooks documentation.

Step-By-Step Instructions

Part 1: Create the ArcGIS Enterprise Portal Webhook 

Create an FME Flow Automation

Before creating a feature service webhook in ArcGIS Enterprise, we first must create a webhook trigger in an FME Flow automation. The webhook trigger will generate a webhook URL, which we can input when creating the webhook on our feature service.

1. Open FME Flow and Create a New Automation
In the FME Flow Sidebar, click Automations > Create Automation.


2. Configure the Webhook Trigger
On the automation canvas, click on the green Trigger icon to open the parameters. For the Trigger parameter, set it to Webhook (triggered) and then click Apply.


Assign a new name to the automation and click Save.
Once saved, click on the Webhook Trigger to open the parameters again. Now you should see a Webhook URL has been populated that we can use for our feature service webhook.



3. Start the Automation
To allow access to the webhook URL, start the automation. This is a requirement for creating the webhook on your feature service.
Create a Feature Service in ArcGIS Enterprise
If you haven’t already, publish the shapefile provided with this article to your ArcGIS Enterprise instance to follow along with the rest of the steps. Alternatively, you can follow along with the next steps using your own feature service. 

Create a Feature Service Webhook

1. Log in to your ArcGIS Enterprise Administrator Directory

Feature service webhooks are created in the ArcGIS Server Administrator Directory.
Log in to your ArcGIS Enterprise Administrator Directory. Your URL should look like the following: https://your-portal-url/arcgis/admin/services

For example, if our portal instance is https://safe-ville.com/portal, then we will log in using the following URL: https://safe-ville.com/arcgis/admin/services

Click on the Hosted folder.


Then click on the feature service you want to create a webhook on. In this case, we published our Public_Art shapefile to the City_Art feature service. 
On the next page, scroll to the bottom and click on the webhooks resource.



In the next page, click on the create button to create a new webhook. 


If you encounter an Error message stating Feature Service must have Editing and Change Tracking enabled without Sync, then you need to go back to your Feature Layer settings and enable “Keep track of changes to the data (add, update, delete features).


In the Create Webhook dialog, set the parameters as follows:

• Name: FME Flow Webhook
• Change Types: FeaturesCreated, FeaturesUpdated, FeaturesDeleted
• HookUrl: This is the URL from your Webhook Trigger.

The rest of the parameters can be left as the defaults. Click Create.


If you get an Error message stating that your URL is not reachable, then you need to start your FME Flow Automation first before attempting to create the webhook.

Once your webhook is created, you should see it listed under your Webhooks. Make sure that Active = true. If it does not, click on the Webhook Id and then click edit to set Active to True.
Your webhook is now set up and ready to use in the FME Flow Automation.
 

Part 2: Create a Workspace to Process the Webhook Payload

In this part, we’ll create a workspace in FME Workbench that can be run by the FME Flow automation when the webhook triggered is set off. The workspace will parse the JSON webhook message and provide the attributes to an FME Flow Automation writer so that they can be included in an email sent from the Automation.

1. Create a Workspace
Open FME Workbench and create a new workspace. Save the workspace as featureservicewebhook.fmw.

In the Navigator pane, right-click User Parameters and select Manager User parameters. 
Click the Insert (+) button and choose Text.


Set the parameters as follows:

• Parameter Identifier: webhook_payload
• Prompt: Webhook Payload

Leave the rest of the parameters as the defaults and press OK to close the User Parameter Manager.


2. Add a Creator
Add a Creator transformer to your workspace. The Creator transformer will kick off the workflow. Next, we need to expose the webhook_payload user parameter into an attribute so we can use it in a transformer that will parse the webhook payload. To do so, add a ParameterFetcher to the workspace and connect the Creator to it. 

Open the ParameterFetcher parameter, click on the red box under Parameter Name and click the drop-down arrow to select webhook_payload. Accept the default Target Attribute name that is automatically populated. Press OK to accept the parameters.


3. Add an ArcGISPortalWebhookDataGetter
To further extract information from the webhook message received from the ArcGIS Enterprise feature service webhook, we will need to use a transformer called the ArcGISPortalWebhookDataGetter.

Add an ArcGISPortalWebhookDataGetter to your workspace. This transformer is available on the FME Hub. You can also begin typing on your empty canvas to search for the transformer. Selecting the transformer should download it if you have not already downloaded it. By default, the transformer will be embedded, which makes it easier for publishing to FME Flow later.


Connect the ParameterFetcher Output port to the ArcGISPortalWebhookDataGetter. 

Open the ArcGISPortalWebhookDataGetter parameters and set the parameters as follows:

• Webhook Response Message: Attribute Value > _webhook_payload
• Select a Web Connection: Click the drop-down arrow > User Parameter > Create User Parameter. Accept the default values and set the Default Value to your ArcGIS Portal web connection. This is the same ArcGIS Portal where you created your feature service webhook.
  • If you have not created a connection, please see How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0)

The User Parameter will allow you to select a verified web connection in FME Flow.

4. Parse the JSON
The response from the ArcGISPortalWebhookDataGetter will be a JSON response that contains information about the newly created feature, such as the user who created it, and the attributes of the feature itself. To extract this information from the JSON response, we can use a JSONExtractor transformer. To learn more about how to work with JSON in FME, see our Getting Started with JSON tutorial. 

Add a JSONExtractor to the workspace and connect the ArcGISPortalWebhookDataGetter Adds port to it. We will use the JSONExtractor to extract the geometry as well as the attributes.
Open the transformer parameters and configure them as follows:

• JSON Document: click the drop-down arrow to select the _response_body attribute
• Extract Queries
  • Target Attribute: X | JSON Query: json["geometry"]["x"]
  • Target Attribute: Y | JSON Query: json["geometry"]["y"]
  • Target Attribute: Attributes | JSON Query: json["attributes"]

5. Parse for the Attributes and their Values
Now that we have an attribute that contains attribute:value pairs in the form of JSON, we need to parse this into attributes.

Add a JSONFragmenter transformer to the workspace and connect the JSONExtractor Evaluated output port to it. 
Open the parameters and configure them as follows:

• JSON Attribute: _response_body
• JSON Query: json[“attributes”]
• Flatten Query Result into Attributes: Yes

Since we know what feature service layer we have set up the webhook on, we can specify the attributes we want to retrieve from the JSONFragmenter in the Attributes to Expose parameter. Click on the ellipsis button (...) and specify the attributes. 
Alternatively, you can click the Import button and import the attribute names from the ArcGIS Portal feature service directly.

For the rest of the parameters, we can accept the defaults. Press OK to close the transformer parameters.

In the example above, we have specified the attributes based on the ones that our feature service contains.


6. Convert Datetime Values to a Readable Format
The datetime format of the last_edited_date attribute is in a format that’s not human-readable upon a quick glance. Since we will be including the last_edited_date attribute in our update email in our FME Flow automation, we will need to convert these values from Epoch time to ISO datetime.

Add an ExpressionEvaluator to the workspace and connect it to the JSONFragmenter. Open the transformer parameters. Change the Evaluation mode to Create New Attribute. For Attributes To Overwrite, select the created_date attribute. In the Arithmetic Expression text box, enter the following: 

@Value(last_edited_date)/1000 

Press OK to accept the parameters.


Add a DateTimeConverter to the workspace and connect it to the ExpressionEvaluator. Set the parameters as follows:

• Datetime Attributes: last_edited_date
• Input Format: %s$ (Epoch Time)
• Output Format: %Y-%m-%dT%H:%M:%S (ISO datetime)

Using these two transformers, we have transformed the last_edited_date from 1689026939978.4534 (Epoch Time) -> 2023-07-10T22:08:59.9784534 (ISO Datetime).

7. Create Geometry
Using the X and Y attributes from the JSON response we parsed earlier, we can create 2D geometry. Add a VertexCreator to the workspace and connect the DateTimeConverter Output port to it. Open the VertexCreator parameters and configure as follows:

• X Value: click the drop-down arrow > Attribute Value > X
• Y Value: click the drop-down arrow > Attribute Value > Y

Press OK to accept the new parameters.

8. Set the Coordinate System
Since we just created the geometry using X and Y values, FME does not know what coordinate system the geometry uses, so we need to assign that ourselves. Fortunately, we know that ArcGIS Portal Feature Service webhooks return features using Spherical Mercator.

Add a CoordinateSystemSetter to the workspace and connect it to the VertexCreator Output port. Open the transformer parameters and configure as follows:

• Coordinate System: SPHERICAL_MERCATOR

Press OK to accept the new parameters.

9. Add an AttributeRemover
The last step before we upload the data to our destination table is to clean up our attributes by removing any unwanted attributes. 

Add an AttributeRemover to your workspace and connect it to the CoordinateSystemSetter Output port. Set the parameters as follows: 

• Attributes to Remove: 
  • _creation_instance
  • _response_body
  • _webhook_payload
  • json_index
  • json_type
  • Attributes
  • X
  • Y

10. Add a SpatiaLite Writer
Now that we’ve formatted the data, we are ready to upload the features to a new SpatiaLite database.

Add a writer to the workspace and set the format to SpatiaLite. Set the Dataset to the location where you want to output your new database. Name the database City_Art.sqlite. Set the Table Definition to Automatic then click OK to add the writer to the workspace.

In the Writer Feature Type window that appears, set the Table Name to Public_Art. Set the Geometry to db_point.


Press OK to close the Writer Feature Type window. Now, connect the AttributeRemover Output port to the SpatiaLite Writer. You will see the attributes on the writer automatically populate. We are now ready to publish our workspace to FME Flow.

11. Publish the Workspace to FME Flow
Save your workspace. Your final workspace should look like the screenshot below:


In FME Form, click the Publish button. Select your FME Flow Connection, Repository and assign a Workspace Name. We will use the same name as our workspace - featureservicewebhook.fmw. 


Click Next. In the Upload Connections dialog, select your ArcGIS Portal connection to ensure it is uploaded to your FME Flow instance. If you already have a web connection defined in FME Flow, then you can skip this step.


Click Next. In the Register Services dialog, we only need the Job Submitter Service so we can accept the default. Click Publish to upload your workspace to FME Flow.

Part 3: Completing the FME Flow Automation

With Parts 1 & 2 completed, we now have a feature service enabled webhook that is ready to be triggered, and a workspace that will process the webhook information.
The final step is to complete our Automation so that it can send the webhook payload to the workspace, and send the corresponding information in an email to alert the employee responsible for managing the feature service.

1. Authorize the ArcGIS Enterprise Portal Web Connection
First things first, the ArcGIS Enterprise Portal web connection that was uploaded when the workspace was published needs to be authorized to work on FME Flow. 

In FME Flow, click Web Connections under Connections & Parameters. From the list of web connections, click the name of your ArcGIS Enterprise Portal web connection.


Click Authorize and allow the request for permission from Esri when prompted. Once authorized, you should see a green check mark beside the connection name.

If the Authorize button is inactive or you receive an error, please ensure you have configured your ArcGIS Enterprise Portal web service correctly in both FME Workbench and FME Flow. Please see: How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0)

2. Add a Run a Workspace Action
Open the Automation you started building in Part 1 of the article series. Click the orange button in the bottom left of the canvas and click on the Action icon to add it to the canvas.


Connect the Webhook Trigger Success Port to the Action. 


Click on the Action icon to open its parameters. Set the parameters as follows:

• Action: Run a Workspace
• Repository: <your repository name>
• Workspace: featureservicewebhook.fmw (or the name you assigned to the workspace when publishing)
• Webhook Payload: Message Content
  • Click the drop-down arrow > Webhook > Message Content
• Select a Web Connection: select your ArcGIS Portal web connection
• SpatiaLite Database: click the three dots and select a location in your Data folder in FME Flow. Make sure to include the name of the database including the .sqlite file extension. For example: "$(FME_SHAREDRESOURCE_DATA)/DanM/City_Art.sqlite"

Once configured, click Apply.


Save your Automation and start it again.

3. Test Run
Let’s test out the Automation to make sure it works. To test out the automation, we can add a new feature to our feature service. 
In your ArcGIS Portal, create a new web map and add the Public_Art layer to it. Save the web map.
Click the Edit button to start an edit session. Under Create features, click on the Public_Art layer.


Click anywhere on the map to create a new feature. Enter values for each of the attributes and then click Create. In the Layers tab, click on the ellipsis next to the Public Art layer and then click Save to save the changes made. When adding a new feature, the webhook is triggered and the payload is sent to FME Flow, which will then process the payload and upload the new features to our destination SpatiaLite database.

In your destination database, we can see that the new feature has been uploaded instantly after changes were made in our ArcGIS Portal feature service, reducing the need to constantly poll the feature service for changes and instead letting ArcGIS Portal & FME Flow do the work.

Your ArcGIS Enterprise Feature Service Webhook Automation is now set up and ready to run! With the automation set up, you can leave it as is or enhance the workflow by adding additional transformers to your workspace to perform analysis or updates when a new feature is added. Using ArcGIS Enterprise Feature Service Webhooks, you can react instantly to changes made to a feature service, all through the power of FME Form & Flow. 
 

Additional Resources

• Configuring FME Flow for HTTPS
• How to create an FME Web Connection for Esri ArcGIS Portal Feature Service (OAuth 2.0)
• Webhooks and FME Flow
• Automate ArcGIS Enterprise Portal Administration with Organization Webhooks and FME Flow
• Integrating ArcGIS Enterprise Geoprocessing Webhooks With FME