Push Data from an ArcGIS Online Feature Service to an Application in Real-time

Introduction

Esri ArcGIS Online Feature Services allow you to serve map features over the web. From mobile apps and workstations, those in the field can have easy access to collect and edit those features. Once edits are made, FME makes it easy to manage your hosted data with the ArcGIS Online Reader/Writer or ArcGISOnlineConnecter. Oftentimes, it’s important to initiate those workflows with new updates and edits made to that Feature Service. For example, we may want to perform the following workflows with an any updated features:

Synchronize the edits with another database
Initiate a process in another application
Send out a notification
Downstream processing

FME Flow (formerly FME Server) Automations allow us to run hands-off workflows with Feature Services. There are two common solutions to initiate an Automation with an updated dataset:

A Schedule Trigger that initiates a workflow on a defined interval. A workspace could then query a Feature Service for changes.
A Webhook Trigger that initiates a workflow as a real-time response to a message received from ArcGIS Online. The webhook message contains information about the changes made to the Feature Service.

Scheduling (1) is a good choice for scenarios that do not require instant updates. FME Flow schedules can be defined on any interval, even with enough frequency that it could approximate real-time updates (But FYI, frequent polling can impact system performance). If you choose this route, consider using change detection to identify any updates made to the Feature Service.

For scenarios where it is critical to respond to events in real-time, we can integrate Feature Service Web Hooks with an FME Flow Automation (2). By leveraging webhooks, we can initiate a workflow within moments of an edit made to a Feature Service.

In the following tutorial, we will set up an FME Flow Automation to receive a message every time a user makes an edit to a Feature Service in ArcGIS Online. This workflow can be used for almost any hosted ArcGIS Online Feature Service that is updated from the web, including most Esri web mapping applications* made with Web App Builder, Instant Apps, and Field Maps.
(*Except Survey123. In this case, use our other tutorial: Automating Workflows from Survey123 to ArcGIS using FME Flow)

Requirements

Your ArcGIS Online account must have permission to use the Admin REST API
FME Flow must be publicly available to receive webhook messages. If you’re using FME Flow Hosted (formerly FME Cloud), no extra configuration is needed.

Step-by-Step Instructions

1. Configure the Automation in FME Flow
Start by building a new Automation in FME Flow that will receive our messages from the ArcGIS Online Feature Service.

Configure the Trigger to Webhook Triggered and select Apply. Attach a Log a Message Action to the Action Succeeded port (the check mark) of the Trigger.

Click the drop-down menu next to the Formatted Message parameter. From the Webhook dropdown, select Message Content. This will let us view the message received from our ArcGIS Online Feature Service in our Automation Log.

2. Get the Webhook URL
Save the Automation and give it a name (E.g. “PlantMapping”). Once saved, double-click the Webhook trigger to reopen its Details. Copy the new Webhook URL to your clipboard. Later in the tutorial, this URL will connect our Automation to the ArcGIS Online Feature Service.

Start the Automation in Debug mode. When you're done building and testing your webhook, you can Start the Automation in Production mode.

3. Create a Hosted Feature Layer for Editing
From your browser, navigate to ArcGIS Online. Select New Item to create a new Feature layer (or proceed with an existing layer from Step 4).

Select Define your own layer. Next, create a Point layer and call it “PlantPoints”.

Finally, add any relevant tags and Save. The PlantPoints feature layer Overview page will automatically open.

From the Feature Service’s Data tab, we can manage any relevant fields for our data. Our employees are mapping invasive plants. So, use the Add button to create new fields for “Species” and "Notes”.

4. Update Settings the Feature Service
From the Feature Layer’s Settings tab, under Editing, enable Keep track of created and updated Features. Save.

5. Enable the Webhook
From the Feature Layer’s Overview tab, scroll to the bottom of the right hand menu to access its REST API URL. Select View to open a new tab containing the PlantPoints "FeatureServer".

Setting up Hosted Feature Service Web Hooks is an REST API Administer request in ArcGIS Online. To access the Admin settings, select "Admin" in the top-right.

Scroll to the bottom of the page and select “Web Hooks”

TIP: If Web Hooks are not available as a supported operation... a) contact your ArcGIS Online administrator to ensure your account has admin privileges, b) ensure the Feature Service is not used in Survey123

6. Create the Webhook
Click Create to set up the webhook for the Feature Service.
Name the webhook.

Change Type specifies what kind of feature layer changes will send messages to our FME Flow Automation. To send messages about new or edited points on our plant layer, we will enter “FeaturesCreated” and “FeaturesUpdated”.

TIP: Many types of events can send messages to an FME Flow Automation. Review “trigger event values” in the Esri documentation for all available options.

In the Hook URL, paste the URL we copied from our FME Flow Automation Webhook Trigger in Step 2.

Payload Format can be PJSON (“Pretty JSON”, for better readability).

Schedule Info can be left as default. Keep in mind that the “interval” key is set to 30. This interval setting hints that our webhook isn’t quite real-time. Instead, ArcGIS Online polls our Feature Layer for changes every 30 seconds. Feel free to increase the polling rate if it’s not critical to get updates ASAP.

Select Create. From the Web Hooks page, select Activate All (leave the default Format as “HTML”).

7. Test the Webhook
Back in ArcGIS Online, add the "PlantPoints" Feature Layer to a new map.

Save the map as "Plant Map".

8. Add a new Point
Because we don’t have any data yet, our map view will be zoomed out to a global scale by default. Zoom into a place of interest, like San Diego.

Open the Edit tab to start editing the layer (if the Edit tab isn’t available, check the feature layer settings in Step 4).

TIP: This is a good time to make sure that your FME Flow Automation has been started in order to receive webhook messages.

From the Create Features menu, select New Feature. Click anywhere on the map to add a new feature. Add some relevant notes. Selecting the Create button finishes the edit and saves the new feature to the Plant Points Feature Service and sends a webhook message to our FME Flow Automation, as will any subsequent edits.

9. View the Webhook message in FME Flow
In FME Flow, open up the Automation we created in Step 1. From the Automation, open up the it's log file (Menu > View Debug Log).

In the log file, there will be a message from our ArcGIS Online Feature Service that corresponds to the new feature we just made to the Plant Points layer.

TIP: If there is no message, try refreshing the page after a 20 second interval. Still no message? Please review the configuration of the Feature Layer, Web Hook, and Web Map covered in Steps 3-8. Also, make sure your FME Flow meets the tutorial requirements.

ArcGIS Online will now send a message to our FME Flow Automation for any new features in the Plant Points Feature Layer. To test getting webhook messages for updated features, navigate back to the PlantMap in ArcGIS Online and modify the new point's attributes. After selecting Update on the dialog, a corresponding message will appear in our FME Flow Automation log for the updated feature.

What next? The webhook message's JSON payload contains a URL (“changesUrl”) that can be accessed to get the details of those changes. Try out using our custom transformer, ArcGISOnlineWebhookDataGetter, to process this message inside a workspace and extract the details of the edited features.