Push Data from ArcGIS Survey123 to an Application in Real-time with Webhooks

Introduction

By combining the power of FME Flow (formerly FME Server) Automations with Esri ArcGIS Survey123 Webhooks, you can extend your field data collection workflows! Inside an automation, a webhook trigger can receive and process Survey123 submissions in real-time. This tutorial configures a webhook in Survey123, builds an automation in FME Flow, and, finally, processes the webhook response. 
 

Requirements

Before starting this guide, we highly recommend that you have completed the FME Form Basic training. If you are new to FME Flow and automations, we also recommend reading our Getting Started with Automations article before continuing. You will need an ArcGIS Online account registered as, minimum, a Creator user type to have access to Survey123. We are assuming you have already set up a survey form in Survey123. You will also need to make your FME Flow publicly accessible if it isn’t already, or, at minimum, anyone submitting the survey must be on the same network as the FME Flow server and have FME Flow configured for HTTPS/SSL.  

Scenario

You are a GIS expert working for an organization that locates and collects data on invasive plant species in the Vancouver area. The organization’s field workers are using smartphones and ArcGIS Survey123 to collect the name of the worker, category of invasive species, location, and a picture of the plant. You have been given the task of using FME Flow to automatically load the Survey123 data and attachments into your organization’s invasive species SpatiaLite database whenever a form is submitted. 
 

Video

Note that this tutorial was made for FME Server 2019 with Notifications and Topics used instead of the webhook trigger.
 

Step-By-Step Instructions

Disclaimer: This tutorial was created using FME Flow 2023.0. There may be some differences in the UI and terminology if you are using a different version, but the fundamentals are mostly the same. Screenshots from ArcGIS Online are also included, but may change without notice.

Before you begin, download the InvasiveSpeciesBegin.fmw workspace as well as the InvasiveSpecies2023.fsproject. (updated for FME 2023.0) from the Files section of this article. Note that both the workspace and automation will need to be configured to your project parameters before they will work.
 

Part 1: Connect the Survey123 Webhook to FME Flow

The first step of creating an automation is setting up the Survey123 form and a webhook trigger to relay the submitted data. For more information on webhooks, please read our Webhooks and FME Flow article.

1. Create Automation in FME Flow
Go to FME Flow. In the side menu, click Automations and select Create Automation.


Double-click the Trigger. In the drop-down menu, select Webhook (triggered). To get a Webhook URL, you will need to save the automation first. Click Apply and save the automation as "Survey123 Invasive Species".

Double-click once again on the Webhook trigger and you should now see the Webhook URL. This URL contains a unique token to give our Survey123 webhook the ability to access FME Flow and run the automation. Leave the parameter open, we will need to copy the Webhook URL in a moment. 


2. Create a Survey123 Form
In a new browser tab, open ArcGIS Survey123. Create a Survey123 form named "Invasive Species Reporting" with a single-line text field for the field worker’s name, drop-down menu of species options, point map, and image attachment. Save and publish the survey.


3. Create a Survey123 Webhook
In the FME Flow automation, copy the Webhook URL from the Webhook trigger by clicking the clipboard button. 

In Survey123, go to the survey settings and click Webhooks in the side menu. Select Add webhook.

Give the webhook a descriptive name so you can remember what it’s being used for in the future. Paste the webhook URL into the Payload URL field. Make sure the New record submitted trigger event and Submitted record event data checkboxes are selected. Click Save to finish.
 

The Survey123 webhook is connected to FME Flow now, and the automation is ready to receive incoming data! However, there are a few more steps required for the automation to properly handle responses from the survey.
 

Part 2: Set Up the Webhook Output Attributes

In this section, we will be creating the webhook output attributes. An output attribute is a message that can be passed through the automation as JSON. Please read Working with Automation Parameters and Custom Attributes in an Automation | User and Global Keys to learn more about output attributes.

So, why do we need output attributes in our automation? The Survey123 webhook sends the submitted form’s attributes, coordinates and other data as JSON parameter-value pairs. These attributes can be referenced in the webhook trigger’s output attributes and passed into our data processing workspace. However, at the moment, we don’t know the structure of the JSON result. We’ll need a message logger to see it.

1. Create a Message Logger
In the Automation, add an Action, connect it to the Webhook trigger Succeeded port, and then set it to Log a message.

Open the drop-down menu next to Formatted Message, expand Webhook, select Message Content, and then click Apply. This will log the entire webhook result message.


2. Submit a Test Form from Survey123
Start the automation in FME Flow. Fill and submit a test form in your survey in ArcGIS Online.

In the automation, open the Menu and click View Log File. In the log, you should see a result similar to the image below.


This is the entire webhook response as JSON; however, we are only interested in the contents of the feature attribute (highlighted). To make the message easier to read, you can paste the message into a text editor or a JSON formatter. If you did not receive the message, verify that your server is publicly visible.

3. Create Webhook Output Attributes
From the log, copy the whole JSON message from the opening curly brace before “applyEdits” to the closing curly brace - do not include the timestamp from the log. Stop the automation, open the Webhook Details, and in Output Attributes select Import Parameters from JSON.

Paste the JSON into the text box and click Parse. 

This will parse through the message and generate output attributes for the top-level JSON parameters.

Click Apply.

In the Log action, you should now be able to select the webhook.feature attribute for the Formatted Message. Select it and click Apply.


Run the automation and submit a test form again. The automation log message will be filtered to just the feature attribute value. Copy this new JSON because we will need it in an upcoming step. Your feature attribute contents should look similar to the code snippet below. The webhook.feature attribute is the only output attribute we’re interested in, since it contains the necessary data to generate a point geometry with all attributes.

{
    "result": {...},
    "attachments": {...},
    "attributes": {
        "what_is_your_name": "Astrid Bennet",
        "select_which_invasive_species_y": "norway_maple",
        "globalid": "{FB29BDB9-79DD-4A7E-882A-A7B5B1932320}",
        "objectid": 57
    },
    "geometry": {
        "x": -122.84153767025904,
        "y": 49.18022516016153,
        "spatialReference": {
            "wkid": 4326
        },
        "geometryType": "esriGeometryPoint"
    },
    "layerInfo": {...}
}

Disclaimer: The webhook output attributes are dependent on the structure of the Survey123 webhook response. Esri could change the webhook structure at any time and your automation output attributes may become invalid.
 

Part 3: Process the Webhook Response JSON in FME Workbench

With the raw data captured and ready to process, we want to connect a workspace as the next action in our automation to convert the webhook JSON into geometry with attributes. In order to pass the webhook.feature output attribute into our workspace, we will need to set up a user parameter in FME Workbench.

1. Create a New Workspace and User Parameters
Open a blank workspace in FME Workbench. Save the workspace as InvasiveSpecies.fmw. In the Navigator, right-click User Parameters and select Manage User Parameters. Click the Insert button and choose Text.

 
On the right-hand side of the Parameter Manager, enter the following parameter properties:

• Parameter Identifier: feature
• Prompt: Webhook Feature
• Published: Enabled
• Required: Enabled
• Disable Attribute Assignment: Disabled
• Default Value: Paste the JSON from the feature attribute response (from the end of Part 2, Step 3). This will be used for testing our workspace. 

Click OK

2. Add a Creator and ParameterFetcher
Search for the Creator transformer and place it on the canvas. This will kickstart the workspace in the automation when the webhook is triggered.

We can grab the webhook.feature output attribute using the ParameterFetcher transformer. Attach a ParameterFetcher transformer to the Creator.

Open the ParameterFetcher parameters. For the Parameter Name, open the drop-down menu and select the feature user parameter that we just created. Name the Target Attribute _webhook.feature. Click OK. 


Run the workspace with Feature Caching Enabled and inspect the ParameterFetcher output. A new attribute _webhook.feature has been created with the JSON text we pasted into the webhook.feature’s default value.


3. Add a JSONFragmenter
In its current state, the _webhook.feature attribute is just text and not functional geometry with attributes. We need to extract the location data and survey attributes and then create a point geometry feature.

Connect a JSONFragmenter to the ParameterFetcher. This transformer will create separate features for each attribute (result, attachments, attributes, geometry, layerinfo).

Set the JSON Attribute to _webhook.feature. In the Flattening Parameters, set Flatten Query Result into Attributes to Yes.
 

4. Filter the JSON Keys
There are three keys we need from the _webhook.feature JSON: attributes, attachments and geometry. 

• Attributes contain the attributes needed for point geometry feature
• Attachments contain the image content type and url needed to download and encode the file
• Geometry contains the information needed, including coordinates, to create the point geometry feature

Add a TestFilter to the canvas and connect it to the JSONFragmenter. Open the TestFilter parameters and add a set of test conditions to filter each of the three keys to different output ports based on the json_index value.


5. Expose and Rename the JSON Attributes
Instead of manually typing the attributes to expose in the JSONFragmenter, we can use the AttributeExposer to import features from the feature cache. Add an AttributeExposer to the workspace and connect the TestFilter’s Attributes, Attachments output ports to it.

Note: If you are using a version of FME older than 2020.1, manually type the attributes into the Attributes to Expose value in the JSONFragmenter.

To use the Import from Feature Cache option, first run the workspace to the TestFilter. 

In the AttributeExposer, click Import > From Feature Cache. Select the Attachments, Attributes, and Unfiltered feature caches and click Next. Select objectId, globalId, what_is_your_name, select_which_invasive_species_y, please_take_a_picture_of_the_sp{}.contentType, and please_take_a_picture_of_the_sp{}.url. The last two attributes are from a list and need a list element specified. Since we’re working with only one image attachment in Survey123, we can specify the first list element: 0. In the list of Attributes To Expose, enter an index of 0 in the curly braces of please_take_a_picture_of_the_sp{}.contentType and please_take_a_picture_of_the_sp{}.url attribute, so that they become please_take_a_picture_of_the_sp{0}.contentType and please_take_a_picture_of_the_sp{0}.url. Your attributes may be different depending on how you labeled them in Survey123. 


The attribute names are not user-friendly, especially the ones that were exposed from the attachments list JSON. Add an AttributeRenamer to the canvas and connect it to the AttributeExposer. Rename the attributes to the following:

• select_which_invasive_species_y: invasive_species_type
• what_is_your_name: name
• please_take_a_picture_of_the_sp{0}.contentType: attachment_type
• please_take_a_picture_of_the_sp{0}.url: attachment_url

6. Aggregate the Attribute Features
We only want one feature with all the attributes we need, so add an Aggregator to the canvas and connect it to the AttributeRenamer. 

Open the Aggregator parameters. Expand the Attribute Accumulation section and set the Accumulation Mode to Merge Incoming Attributes.

Click OK.

Run the workspace and check the Aggregator feature cache. In the Visual Preview, verify that all attributes are present.


The workspace should now look like this:


7. Create a Point Geometry
Now we need to capture the geometry from the JSON and merge it with the attributes from the previous step so that we have one point feature with all the survey information. Add a GeometryReplacer to the TestFilter’s Geometry output port.

In the GeometryReplacer parameters, set the Geometry Encoding to Esri JSON. Enter _webhook.feature for the Geometry Source. 


Add a FeatureMerger to the canvas. Connect the GeometryReplacer output to the FeatureMerger Requestor input port. Connect the Aggregator to the Supplier input port. Open the FeatureMerger parameters. Since we want to merge all attributes from the Aggregator to the new point geometry feature, but we don’t have a common key to join on, we can simply create a join between the Requestor and Supplier where 1 = 1.

 
Run the workspace and inspect the FeatureMerger’s Merged output in Visual Preview. Turn Graphics View on. There is now a point geometry feature with all attributes from the submitted survey, plus the attachment type and URL we’ll need to download the image in the next part.


8. Organize the Workspace
Let’s spend some time organizing our workspace now. Create a bookmark around all of the transformers added so far, except for the Creator and ParameterFetcher. Name it "Create feature geometry from JSON". At this point, your workspace should look like this: 

 

Part 4: Get Image Attachment from ArcGIS Online

We have created point geometry with all attributes included, but we are still missing the image attachment. You may have noticed the image is not included in the webhook response. The image is stored in the ArcGIS Online feature service, therefore we need to download the image from the feature service and add it to the point as a binary attribute so that it can be written into the database.

1. Create an ArcGIS Online Web Connection
In FME Workbench, select Tools > FME Options. In the FME Options dialog box, select Web Connections and add a new web connection.


In the Edit Web Connection dialog, choose Esri ArcGIS Online as the Web Service. Enter a unique Connection Name - this is the name that will appear as a choice when selecting a web connection in the ImageFetcher (and other transformers that can connect via a web service). Click OK. 


When prompted, enter your Esri ArcGIS Online Username and Password and then click Sign In to create the Web Connection.

Please read Web Connections and FME to learn more about web connections in FME.

2. Download the Image from ArcGIS Online
Add an ImageFetcher to the canvas and connect it to the Merged output port of the FeatureMerger. The ImageFetcher will download the image from the URL we obtained from the Attachments key in the feature JSON. Configure the ImageFetcher parameters as follows:

• Image URL: attachment_url
• Image Type: <Auto-Detect from HTTP Headers>
• Use Authentication: Enabled 
• Authentication Method: Named Web Connection
• Web Connection: the name of the ArcGIS Online web connection created in Part 4, Step 1.

Click OK.


Run the workspace up to the ImageFetcher and inspect the results in Visual Preview. In the Graphics View, you should see the image. In this example, some invasive Norway Maple.


3. Test Which Type of Image Needs to be Encoded
The image needs to be encoded as a Blob to be written to the database, but before we can do that, we need to know what type of image we’re encoding. The type can be determined from the attachment_type we obtained from the same Attachments JSON key that contained the URL. For the purposes of this example, we will only handle JPEG and PNG files, but more tests could be added to handle additional types. Add a TestFilter to the canvas and connect it to the ImageFetcher. Create the following tests:

• If @Value(attachment_type) = image/jpeg => send to a ‘JPEG’ output port
• Else If @Value(attachment_type) = image/png => send to a ‘PNG’ output port
• Else => send everything else to an ‘UNSUPPORTED TYPE’ output port

Add a Terminator to the canvas and connect it to the TestFilter UNSUPPORTED TYPE port. Set the Termination Message to: Translation Terminated Due to Unsupported Attachment Type: @Value(attachment_type). Now, when an unexpected attachment type is encountered, the message will be written to the log.

4. Encode the Image
Add a RasterExtractor to the canvas and configure it to encode the image as a JPEG. Connect this RasterExtractor to the TestFilter JPEG port. 


Repeat this process and add another RasterExtractor to encode .png files. After this is complete, your canvas should look like this:


Run the workspace and inspect the RasterExtractor output in Visual Preview. You should see a _rasterBlob attribute and value has been created. This is the raw image data that will be written to the database.


5. Add the Image Blob to the Point Feature
The image blob needs to be added to the point geometry created in Part 3. We could use another FeatureMerger to do this, but since we now have a common attribute, objectid, on both features, we can use a FeatureJoiner. 

Add a FeatureJoiner to the canvas. Connect both RasterExtractors to the FeatureJoiner’s Left input port. Connect the FeatureMerger Merged output port to the Right input port. If the line connecting the transformers is hard to see on the canvas, you can double-click it to add a vertex where you want to reposition the line and then click and drag to move.
Configure the parameters of the FeatureJoiner to join both features on objectid For Geometry Handling, choose Prefer Right, since the right feature contains the point geometry.


6. Remove and Rename Attributes
Now that we have added the encoded image to the point feature, all that’s left is to clean up the attributes before writing to the database.

Add an AttributeManager transformer and connect it to the Joined output port of the FeatureJoiner. Remove all attributes except:

• objectid
• globalid
• name
• invasive_species_type
• attachment_type
• _rasterBlob

Rename _rasterBlob to attachment


Let’s keep our workspace clean and add a bookmark around the transformers between the ImageFetcher and the AttributeManager. Name the bookmark Get the image from ArcGIS Online and encode. The workspace should now look like this:


We are ready to write the output to our database! 

7. Write to a SpatiaLite Database
Add a new writer. Set the format to SpatiaLite, provide an output destination, name the database "InvasiveSpeciesRecords," and set the Table Definition to Automatic. Set the coordinate system to LL-WGS84. Click OK.

In the writer feature type parameters, name the table "survey_results." Choose db_point as the Geometry. Ensure that the Feature Operation is set to Insert and Table Handling to Create If Needed. Click OK. 


Note that a new published parameter was automatically generated named "DestDataset_SpatiaLite". This will be used in our automation to set the output destination in the FME Flow shared resources.

Connect the AttributeManager to the SpatiaLite writer, and run the workspace. View the output data to verify that it was written as expected.

Map tiles by Stamen Design, under CC-BY-3.0. Data by OpenStreetMap, under CC-BY-SA.

You can test the workspace by submitting additional surveys in Survey123 and copying the request JSON from the FME Flow log into the feature parameter, like we did in Part 2, Step 3. Test with a .png file, a .jpeg file, and a different type of file that will terminate the workspace. You can set the writer’s Overwrite Existing Database parameter to Yes while you test (as shown below), but make sure to set it back to No prior to publishing to FME Flow.


8. Publish to FME Flow
Now that our workspace is complete, we need to publish the workspace and the SpatiaLite database to FME Flow. If this is your first time doing this, please read our guide on publishing workspaces to FME Flow before proceeding. It is a good idea to save your workspace before publishing. 

Connect to your FME Flow instance and choose a repository to upload the workspace to (or create a new repository).

We also want to upload the SpatiaLite database. Click 1) Select Files and 2) Add Files. Navigate to your SpatiaLite database and select it. Then click 3) Select Location. Choose to 4) Upload to a shared resource folder. 5)Creating a new folder within Data is advisable. Click OK and Continue.


In the next window, select your ArcGIS Online web connection to upload it to FME Flow. Click Continue. For services, select Job Submitter and click Publish. Check the Translation Log to make sure everything was published to FME Flow successfully.
 

Part 5: Complete the Automation

With our workspace and database in FME Flow, we can add them to our automation now.

1. Authorize the ArcGIS Online Web Connection
First things first, the ArcGIS Online web connection that was uploaded when the workspace was published needs to be authorized. 
In FME Flow, click Web Connections under Connections & Parameters. From the list of web connections, click the name of your ArcGIS Online web connection.


Click Authorize and allow the request for permission from Esri when prompted. Once authorized, you should see a green check mark beside the connection name.


2. Create a Workspace Action
Open the Survey123 Invasive Species automation. Stop the automation if it is still running.

Connect a Log action to the Webhook trigger failed port, instead of the succeeded port. Add a new Action to the canvas and connect the Webhook trigger succeeded port to it. 


Open the Action and set it to Run a workspace. Choose the repository where the InvasiveSpecies.fmw workspace was published. You will notice that the Webhook Feature field is already filled in. This is the test JSON that we used in our workspace’s user parameter default value, but we want the webhook.feature output key instead. Open the drop-down menu, expand the Webhook Attributes, and select webhook.feature.

For the SpatiaLite Database parameter, click the ellipses and navigate to the database file that you uploaded to FME Flow. Select it, click OK, then Apply.


The Survey123 webhook automation is complete and ready to receive form submissions! With this setup, you will be automatically updating your invasive species database whenever a form is submitted, which saves time and effort and streamlines the process. While this workflow writes to a database file that needs to be downloaded to view, you can write to an external database such as PostGIS or SQL Server instead.

This automation is relatively simple, but you can extend your workflow with additional FME Flow and FME Workbench functionality, such as email notifications and generating reports. Please see the Emailing a Survey123 Report Using FME Flow tutorial if you would like to extend this automation.

Note: Remember to remove the test records from your database and ArcGIS Online feature service before you deploy the automation! Also, if you receive an error from the ImageFetcher when your automation runs, for example ‘An image type could not be detected’, it may be an authentication issue due to an out-of-date ArcGIS Online Web Server on your FME Flow. Please see this article for more information: Known Issue: Unable to Authenticate to ArcGIS Online on FME Server.
 

Additional Resources

Getting Started with Automations