Submitting a Job through an FME Flow Webhook URL

Introduction

Webhook URLs allow third-party applications to run jobs on demand. From FME Flow (formerly FME Server), it takes only minutes to create a URL that can access and run a workspace with custom parameters. Share this URL with other applications to submit jobs and deliver web services back to an end-user.

A Webhook URL shares many similarities with job requests made through the REST API. It can run a job synchronously or asynchronously. It may include directives for job control, use custom datasets, and support token authentication. But instead of configuring these options inside a POST request, a webhook includes all this information directly in the URL.

Workspaces run via Webhook URL can access all the FME Flow Transformation Services, including the Job Submitter, Data Streaming, and Data Download services. This makes them a great solution for dynamic and responsive web applications. Conversely, jobs run by REST API are limited to the REST Service.

Consider your applications, systems, and workflow when deciding whether to submit jobs using Webhook URLs or the REST API. You might even find that both methods benefit your workflow.

In this tutorial, we’ll create a Webhook URL that runs a workspace. We’ll use this URL to submit a job using Postman and a standard browser. Please keep in mind that applications outside of your network must submit requests over the web, so please ensure that your FME Flow instance is publicly available.

Files

Download the required files and workspace from S3.

Neighbourhoods.sqlite

Vancouver.tif

VanImagerySelector.fmwt

Step-by-Step Instructions

Many organizations allow others to access their datasets through websites, portals, or custom applications. Webhook URLs can provide a simple solution for making these requests.

In this step-by-step tutorial, we will demonstrate submitting jobs by Webhook URLs. The result is a URL that allows an application to request custom satellite imagery. This data can be requested by neighborhood name, and results can be received in different ways.

Part 1 prepares and uploads the workspace to FME Flow. Parts 2 and 3 present two scenarios that demonstrate the versatility of Webhook URLs and the FME Flow Transformation Services. We’ll first make the request from a browser and view the results in the same window using the Data Streaming service (Part 2). Then, we’ll make the request from Postman and have the results returned as a downloadable file using the Data Download service (Part 3).

Part 1: Publish the Workspace to FME Flow

1. Configure the Workspace in FME Workbench
Open the VanImagerySelector.fmwt workspace in FME Workbench.

We can control what the client (our user or application) will have access to change or not by carefully considering the workspace’s User Parameters. Any published parameters can be accessed and modified from a webhook URL.

Our FEATURE_TYPES parameter determines which tables are read from the Spatialite database. Each table contains a polygon of a different Vancouver neighborhood. Our webhook will allow the client to choose which neighborhoods they’d like to view imagery for.



Whenever possible, run the translation from FME Workbench to make sure everything is working correctly. When all the Feature Types to Read are selected, the workspace writes a PNG Raster image (NeighbourhoodSat.png) clipped to the City of Vancouver’s boundaries. Please note that to run this workspace from Form successfully, you must set local file paths for all the readers and writers.



2. Publish the Workspace to FME Flow
From the FME Workbench ribbon menu, select the Publish button to start the Publishing Wizard.

After connecting to your FME Flow, proceed to create a new repository (New...) and name it “VanImageryApp”. Make sure to select the data files (Neighbourhoods.sqlite and Vancouver.tif) to upload. Continue.



3. Register Web Services
FME Flow Services allow us to control how the end user receives the translation results. Select Data Download, Data Streaming, and Job Submitter. Publish.

Part 2: Webhook URLs and the Data Streaming Service

Webhook URLs can be used with the Data Streaming Service to instantly display data back to a user. This is a great solution for applications that create visualizations, maps, charts, or graphics for supported formats.

In Part 2, we will create a Webhook URL for the VanImagerySelector.fmw workspace (available in the Files section). The resulting workflow streams customized satellite imagery back to their browser.

1. Configure the Workspace in FME Flow
Open FME Flow in a browser. Select Run Workspace from the sidebar.

Select the “VanImageryApp” Repository, “VanImagerySelector.fmw” Workspace, and “Data Streaming” Service. Leave the default values for Feature Types to Read.



2. Create the Webhook URL
Select the Advanced panel to expand. Scroll down to the section "Other Ways to Run this Workspace," which lists options for making workspaces self-serve.

Select Create a Webhook.



3. Configure the Webhook URL and API Token
The Share page allows us to preview our final API Token and Webhook URL.

FME Flow automatically creates a new API token for every Webhook URL. The API token allows another application to run the workspace and any other workspaces saved to the same repository without an FME Flow account. Permissions are automatically restricted to the workspace repository and any dependencies, but this may be modified after creation from the Manage Tokens page.

The Webhook URL Preview contains all the required information to run a particular workspace from the web: the FME Flow domain, web service, repository, and workspace. Finally, any parameters and their values are followed by a query string.

To modify the default URL, expand the Parameters panel. Try removing one of the values (e.g., “Downtown”) and watch how it changes Webhook URL Preview in real-time.

If the workspace uses a geometry parameter, the webhook URL will need to be URL-encoded geojson.



We can also preview what our Webhook URL results will look like. Select the clipboard next to the Webhook URL Preview to copy the link. Open a new browser tab and paste the link into the search bar. (You may be prompted to log into FME Flow again. This is because our Webhook URL Preview link does not yet include authentication).

View our clipped PNG Raster now from a browser window. If you removed any values from the Feature Types to Read parameter, your raster may be clipped differently than the one pictured below.



Back on the FME Flow Share page, select Reset to restore Parameter default values. Select OK.

4. Save the Webhook
Before doing anything else, select the Download Webhook button to save a record to your desktop. This file contains information about our new Webhook URL, and it is the only opportunity we have to save its newly created token.

To authenticate a job run by Webhook URL, a token is placed in the headers or query string parameters. View examples of each on this page (or inside the saved file). In most cases, headers are used for the most secure and reliable approach. For our example, select the clipboard next to the Authorization with the Query String link to copy.

5. Submit a Job by Webhook URL
Open another browser window in incognito mode to test our new token. Paste the Webhook URL into the search bar and submit. Our PNG Raster, with all the neighborhoods selected, will be streamed back. But this time, we didn’t have to log in! Try sharing this link with a trusted colleague or friend; they can get results even without an FME Flow account.

6. Submit a Job by Webhook URL with Updated Parameters
Webhook URLs can use different parameter values at runtime. Next to the query string’s FEATURE_TYPES parameter, delete all the values except “Downtown” (including the URL encoding). The result will look like this URL:

Submit to view the updated result. This time, the output PNG Raster is clipped only to the “Downtown” selection.



Configurable parameters are especially useful for web applications that can create new requests programmatically.

If you are using geometry parameters, the webhook URL will only work if you are using URL-encoded geojson. If you re-select the geometry parameter on the Create Webhook page, then the URL is updated with the correct URL-encoded FME Parsable Text.

7. View the Jobs in FME Flow
In FME Flow, navigate to the Jobs > Completed page. Find our last submitted job by the workspace name and time of submission. Keep in mind that the Username and Ran By columns will be populated by the user who created the token used with the Webhook URL, regardless of the application that was submitted.



Select the job to review the log. Expand REQUEST DATA to find the parameters.



Scroll down the log to find the feature summaries. The Features Read Summary reflects the selected input data (Selected Spatialite tables and the GeoTIFF imagery). The Features Written Summary is the streamed output data (the clipped PNG Raster).



Congratulations! You’ve now run a job using multiple parameters via Webhook URL.

Part 3: Webhook URLs and the Data Download Service

Webhook URLs can be used with the Data Download Service to deliver a downloadable file with job results. A common application of this workflow might be a data portal where users can request and download custom datasets on demand.

In Part 3, we will create a Webhook URL that runs the same VanImagerySelector.fmw workspace. However, this time, it returns a zip file containing the custom imagery. Instead of making the request from a browser, we’ll use Postman to submit the final request. Finally, we will automatically create a Python script to execute a webhook URL. These exercises are meant to demonstrate how FME jobs can be easily run from other applications, including third-party software or custom-coded.

Part 3 of this tutorial builds off the lessons and concepts learned in Parts 1 and 2.

1. Configure the Workspace in FME Flow
Open FME Flow in a browser. Select Run Workspace from the sidebar.

Select the “Webhook URL Tutorial” Repository, “VanImagerySelector.fmw” Workspace. This time, select “Data Download” Service. Leave the default values Feature Types to Read.


Select the Advanced panel to expand. Scroll down to find the section Other Ways to Run this Workspace and select Create a Webhook.

2. Set Webhook URL Parameters
The URL preview and API token are almost identical to that in Part 2, but there is one key difference: this time, our URL specifies the fmedatadownload service. Select the clipboard next to the Webhook URL Preview to copy the link.

Open another browser tab and paste the link in the search bar (logging in again, if prompted).

A page containing a link with our results is returned.



Back on the FME Flow Share page, select Reset to restore Parameter default values if needed. Select OK.

3. Create a Token
Select the Download Webhook button to save a record of this information to your desktop.

Select the clipboard next to the Authorization with Header link to copy.



4. Submit a Job in Postman
In Postman, open a new tab to begin building the request. Paste the URL into the space provided in Postman. Use the dropdown menu to change the method from GET to POST.



You may notice that the request contains additional query parameters: opt_showresult and opt_servicemode. These are examples of parameters specific to the data download service. Each transformation service has parameters that control how to run the job and deliver the service. For this request, opt_showresult=false indicates that the full translation details should not be returned in the response, and opt_servicemode=sync submits a synchronous job.

Add an additional header to format the response in a JSON: opt_responseformat=json


5. Create Headers
Passing tokens as a request header is the preferred method of authentication.

In FME Flow, copy the Header Value given under the Authorization with Header URL.

In the Postman request, select the Headers tab. Create a new header with Authorization. Paste the copied value as the VALUE.



Once the header is completed, select SEND to submit the request.

6. Get the Results
A successful request returns a link to the results compressed in a zip file. Subsequent requests can use this link to download the results.


Copy the value given for the “url” and paste it into a new browser window. Instead of a results page, you’ll immediately be prompted to download a zip file to your local desktop.

Extract the contents to reveal the requested PNG raster. View in any image editor application, browser, or the FME Data Inspector.



Like our previous requests, we can review the translation details in FME Flow. To find the job log, navigate to your FME Flow Completed Jobs page.

7. Use Webhook URLs in Code
Webhook URLs, like REST API, are a great integration tool for scripted applications.

In Postman, select the “</>” button from the left-hand sidebar to open the Code snippet tool. Select any of the options from the dropdown menu to generate code for your Webhook URL request. For example, selecting “Python - Requests” produces Python code that runs a workspace on FME Flow:



Try to execute this code from your preferred compiler to run a workspace in FME Flow.

Congratulations! You’ve now sent a job request via an FME Flow Webhook URL from a third-party application.

Troubleshooting

Failed Requests and Error Codes

The FME Flow REST API returns an error when it rejects a request. The API response contains the error’s code and message.

For an overview of general error codes and their corresponding messages, review the HTTP Response Codes and Errors section of the FME Flow REST API documentation.

Error messages can also be specific to the request type. For example, a job request may be rejected when a published parameter is invalid. Select a request in the documentation to view the Response Status Codes:

Authentication Errors

When a token is rejected, you’ll receive either a “401 Unauthorized” or “403 Forbidden” error code.

Tokens are likely to be rejected when they expire or if permissions are lacking for workflow dependencies. For authentication options, please review the Authorization section of the FME Flow REST API documentation.

Additional Resources

Article: Submitting a Job through the FME Flow REST API
Article: Token Management in FME Flow
Article: Getting Started with the FME Flow REST API