Submitting a Job through an FME Flow Webhook URL

Introduction

Webhook URLs allow third-party applications to run jobs on demand. From FME Flow (formerly FME Server), it takes only minutes to create a URL that can access and run a workspace with custom parameters. Share this URL with other applications to submit jobs and deliver web services back to an end-user.

In many ways, a Webhook URL shares a lot in common with job requests made through the REST API. A Webhook URL can run a job synchronously or asynchronously. It may include directives for job control, use custom datasets, and supports token authentication. But instead of configuring these options inside a POST request, a webhook includes all this information directly in the URL.

Workspaces run via Webhook URL can access all the FME Flow Transformation Services, including the Job Submitter, Data Streaming, and Data Download services. This makes them a great solution for dynamic and responsive web applications. Conversely, jobs run by REST API are limited to the REST Service. 

Consider your applications, systems, and workflow when deciding whether to submit jobs by Webhook URLs or the REST API. You might even find that both methods have a role to play in your workflow. 

In this tutorial, we’ll create a Webhook URL that runs a workspace. We’ll use this URL to submit a job by Postman and a standard browser. Please keep in mind, in order for applications outside of your network to submit requests over the web, please ensure that your FME Flow instance is publicly available. 

Files

Download the required files and workspace from S3. 

Neighbourhoods.sqlite

Vancouver.tif

VanImagerySelector.fmwt

Step-by-Step Instructions

Many organizations allow others to access their datasets through websites, portals, or custom applications. Webhook URLs can provide a simple solution for making these requests. 

In this step-by-step tutorial, we will demonstrate submitting jobs by Webhook URLs. The result is a URL that allows an application to request custom satellite imagery. This data can be requested by neighbourhood and results can be received in different ways. 

Part 1 prepares and uploads the workspace to FME Flow. Parts 2 and 3 present two scenarios that demonstrate the versatility of Webhook URLs and the FME Flow Transformation Services. We’ll first make the request from a browser and view the results in the same window using the Data Streaming service (Part 2). Then, we’ll make the request from Postman and have the results returned as a downloadable file using the Data Download service (Part 3). 

Part 1: Publish the Workspace to FME Flow

1. Configure the Workspace in FME Workbench
Open the VanImagerySelector.fmwt workspace in FME Workbench. 

We can control what the client (our user or application) will have access to change, or not, by careful consideration of the workspace’s User Parameters. Any published parameters can be accessed and modified from a webhook URL. 

Our FEATURE_TYPES parameter determines which tables are read from the Spatialite database. Each table contains a polygon of a different Vancouver neighbourhood. Our webhook will allow the client to choose which neighbourhoods they’d like to view imagery for.


 
Whenever possible, run the translation from FME Workbench to make sure everything is working correctly. When all the Feature Types to Read are selected, the workspace writes a PNG Raster image (NeighbourhoodSat.png) clipped to the City of Vancouver’s boundaries. Please note that to successfully run this workspace from Form, please set local file paths for all the Readers and Writers.


 
2. Publish the Workspace to FME Flow
From the FME Workbench ribbon menu, select the Publish button to start the Publishing Wizard. 

After connecting to your FME Flow, proceed to create a new repository (New...) and name it “VanImageryApp”. Make sure to select the data files (Neighbourhoods.sqlite and Vancouver.tif) to upload. Continue. 


 
3. Register Web Services
FME Flow Services allow us to control how the end user receives the translation results. Select Data Download, Data Streaming, and Job Submitter. Publish. 

Part 2: Webhook URLs and the Data Streaming Service

Webhook URLs can be used with the Data Streaming Service to instantly display data back to a user. This is a great solution for applications that create visualisations, maps, charts, or graphics for supported formats.  

In Part 2, we will create a Webhook URL for the VanImagerySelector.fmw workspace (available in the Files section). The resulting workflow streams customised satellite imagery back to their browser. 
 
1. Configure the Workspace in FME Flow
Open FME Flow in a browser. Select Run Workspace from the sidebar. 

Select the “VanImageryApp” Repository, “VanImagerySelector.fmw” Workspace, and “Data Streaming” Service. Leave the default values for Feature Types to Read. 


 
2. Create the Webhook URL
Select the Advanced panel to expand. Scroll down to find the section Other Ways to Run this Workspace which lists options to make workspaces self-serve. 

Select Create a Webhook. 


 
3. Configure the Webhook URL and API Token
The Share page allows us to preview our final API Token and Webhook URL. 

FME Flow automatically creates a new API token for every Webhook URL. The API token allows another application to run the workspace, and any other workspaces saved to the same repository, without an FME Flow account. Permissions are automatically restricted to the workspace repository and any dependencies, but this may be modified after creation from the Manage Tokens page.

The Webhook URL Preview contains all the required information to run a particular workspace from the web: the FME Flow domain, web service, repository, and workspace. Finally, any parameters and their values follow in a query string. 

To modify the default URL, expand the Parameters panel. Try removing one of the values (e.g. “Downtown”) and watch how it changes Webhook URL Preview in real-time.


 
We can also preview what our Webhook URL results will look like. Select the clipboard next to the Webhook URL Preview to copy the link. Open a new browser tab and paste the link into the search bar. (You may be prompted to log into FME Flow again. This is because our Webhook URL Preview link does not yet include authentication).

View our clipped PNG Raster, now from a browser window. If you removed any values from the Feature Types to Read parameter your raster may be clipped differently than pictured below.


 
Back on the FME Flow Share page, select Reset to restore Parameter default values. Select OK. 
 
4. Save the Webhook
Before doing anything else, select the Download Webhook button to save a record to your desktop. This file contains information about our new Webhook URL. It is the only opportunity we get to save its newly created token.

To authenticate a job run by Webhook URL, a token is placed in the headers or query string parameters. View examples of each on this page (or inside the saved file). In most cases, opt to use headers for the most secure and reliable approach. But for our example, select the clipboard next to the Authorization with Query String link to copy. 

 
5. Submit a Job by Webhook URL
Open another browser window in incognito mode in order to test our new token. Paste the Webhook URL into the search bar and submit. Streamed back is our PNG Raster, with all the neighbourhoods selected. But this time, we didn’t have to log in! Try sharing this link with a trusted colleague or friend, they can get results even without an FME Flow account. 


 

 
6. Submit a Job by Webhook URL with Updated Parameters
Webhook URLs can use different parameter values at runtime. Next to the query string’s FEATURE_TYPES parameter, delete all the values except “Downtown” (including the URL encoding). The result will look like this URL:

Submit to view the updated result. This time, the output PNG Raster is clipped only to the “Downtown” selection. 



Configurable parameters are especially useful for web applications that can create new requests programmatically.
 
7. View the Jobs in FME Flow
In FME Flow, navigate to the Jobs > Completed page. Find our last submitted job by the workspace name and time of submission. Keep in mind that the Username and Ran By columns will be populated by the user that created the token used with the Webhook URL, regardless of the application that submitted it. 



Select the job to review the log. Expand REQUEST DATA to find the parameters. 



Scroll down the log to find the feature summaries. The Features Read Summary reflects the selected input data (Selected Spatialite tables and the GEOTIFF imagery). The Features Written Summary is the streamed output data (the clipped PNG Raster). 


 
Congratulations! You’ve now ran a job, using multiple parameters, via Webhook URL. 

Part 3: Webhook URLs and the Data Download Service

Webhook URLs can be used with the Data Download Service to deliver a downloadable file with job results. A common application of this workflow might be a data portal where users can request and download custom datasets on-demand. 

In Part 3, we will create a Webhook URL that runs the same VanImagerySelector.fmw workspace. However, this time it returns a zip file containing the custom imagery. Instead of making the request from a browser, we’ll use Postman to submit the final request. Finally, we will automatically create a Python script to execute a webhook URL. These exercises are meant to demonstrate how FME jobs can be easily run from other applications, including third-party software or custom-coded.
 
Part 3 of this tutorial builds off the lessons and concepts learned in Parts 1 and 2.
 
1. Configure the Workspace in FME Flow
Open FME Flow in a browser. Select Run Workspace from the sidebar. 

Select the “Webhook URL Tutorial” Repository, “VanImagerySelector.fmw” Workspace. This time, select “Data Download” Service. Leave the default values Feature Types to Read. 

 
Select the Advanced panel to expand. Scroll down to find the section Other Ways to Run this Workspace and select Create a Webhook. 

2. Set Webhook URL Parameters 
The URL preview and API token are almost identical to that in Part 2 with one key difference: this time, our URL specifies the fmedatadownload service. Select the clipboard next to the Webhook URL Preview to copy the link. 

Open another browser tab and paste the link in the search bar (logging in again, if prompted).  

A page containing a link with our results is returned. 


 
Back on the FME Flow Share page, select Reset to restore Parameter default values if needed. Select OK. 
 
3. Create a Token
Select the Download Webhook button to save a record of this information to your desktop. 

Select the clipboard next to the Authorization with Header link to copy. 


 
4. Submit a Job in Postman
In Postman, open a new tab to begin building the request. Paste the URL into the space provided in Postman. Change the method from GET to POST using the dropdown menu. 


 
You may notice that the request contains additional query parameters: opt_showresult and opt_servicemode. These are examples of parameters specific to the data download service. Each transformation service has parameters that control how to run the job and deliver the service. For this request, opt_showresult=false indicates not to return the full translation details in the response and opt_servicemode=sync submits a synchronous job. 

Add an additional header to format the response in a JSON: opt_responseformat=json 

 
5. Create Headers
Passing tokens as a request header is the preferred method of authentication.

In FME Flow, copy the Header Value given under the Authorization with Header URL. 

In the Postman request, select the Headers tab. Create a new header with Authorization. Paste the copied value as the VALUE. 



Once the header is completed, select SEND to submit the request. 
 
6. Get the Results
A successful request returns a link to the results compressed in a zip file. This link can be used in subsequent requests to download the results. 

 
Copy the value given for the “url” and paste it into a new browser window. Instead of a results page, you’ll immediately be prompted to download a zip file to your local desktop.

Extract the contents to reveal the requested PNG raster. View in any image editor application, browser, or the FME Data Inspector.


 
Like our previous requests, we can review the translation details in FME Flow. Navigate to your FME Flow Completed Jobs page to find the job log. 
 
7. Use Webhook URLs in Code
Webhook URLs, like REST API, are a great integration tool for scripted applications.

In Postman, select the “</>” button from the left-hand sidebar to open the Code snippet tool. Select any of the options from the dropdown menu to generate code for your Webhook URL request. For example, selecting “Python - Requests” produces python code that runs a workspace on FME Flow:



Try to execute this code from your preferred compiler to run a workspace in FME Flow. 

Congratulations! You’ve now sent a job request via an FME Flow Webhook URL from a third-party application.

Troubleshooting 

Failed Requests and Error Codes 

The FME Flow REST API returns an error when it rejects a request. The API response contains the error’s code and message. 

Review the HTTP Response Codes and Errors section of the FME Flow REST API documentation for an overview of general error codes and their corresponding messages.

Error messages can also be specific to the request type. For example, a job request may be rejected when a published parameter is invalid. Select a request in the documentation to view the Response Status Codes:

Authentication Errors

When a token is rejected, you’ll receive either a “401 Unauthorized” or “403 Forbidden” error code. 

Tokens are likely to be rejected when they expire or do not have permissions to the workflow dependencies. Please review or the Authorization section of the FME Flow REST API documentation for authentication options. 

Additional Resources

Article: Submitting a Job through the FME Flow REST API
Article: Token Management in FME Flow
Article: Getting Started with the FME Flow REST API