Send Source Data to FME Server with HTTP POST

Introduction

To support real-time data applications, you may want to POST data to FME Server and trigger a workspace that reads this data as a source dataset. For example, you can create triggers for sensors that can send sensor data to an FME Workspace for processing or you could create your own application that submits form data to FME Server.

Please Note:

• Using the method described in this article, FME Server accepts raw data in the POST body only, but does not accept form-data. Form-data can be used to submit workspace parameters simply by sending parameter names and values as form element names and values.
• Currently, this method is best for simple text data such as XML or JSON and does not support multipart file upload. If you want to upload file data to be used as the source dataset for an FME Workspace, please see the article, Data Upload with FME Server. 

The following Web Services can accept raw data sent to FME Server using HTTP POST:

• FME Data Download
• FME Data Streaming
• FME Job Submitter

You will be using FME Server REST API and creating your own application to solve data challenges and ultimately to leverage the power of FME. A REST API allows a user to interact with the server by creating requests and receiving responses using HTTP Protocols. Requests are made through URLs and responses are formatted in JSON. HTTP Methods are essential to every call to the REST API. In other words, HTTP Methods represent an action to FME Server that you would like to complete. The HTTP Method includes: GET, POST, PUT, or DELETE. We will be focusing on the HTTP POST request, which is used to create a publication, project, or notification. You may follow the FME Server 2019 REST API Training for an in-depth understanding of FME Server REST API.

Visit the FME Server REST API page to learn more about the different actions and requests that are available. You can also get to this page via your FME Server web interface in the help icon, REST API tab.
 

Postman is a REST Client Tool to make calls to FME Server. If there is another REST client tool you are familiar with, you may use that as an alternative. The instructions in this article will refer to the Postman interface.

Webhooks are URLs that allow external software to run the workspaces. To learn more about webhooks and FME Server, read the Webhooks and FME Server article. 

 

Requirements

• Download Postman
• Sample.zip (available from the Files section on this article

Step-by-step Instructions

Prepare the Data and Create a Workspace

1. Create a simple text file that is structured to represent a sample of data that you will be sending (e.g. an XML or JSON snippet). In the “Samples” folder attached to this tutorial, there is a very simple data file called Sample.txt that may be used. This text file will be called later on when posting to FME Server with the Data Streaming service.

2. Build an FME workspace that will read the data you POST to it. In most cases, using the Text File Reader is best. Even if the data is XML or JSON, you can still use the Text File reader. Make sure to set the reader parameter "Read Whole File at Once" to Yes. You can then parse the data further along in the workspace using the appropriate XML or JSON transformers.

3. Add a writer to the workspace for output data - which will be sent to the Data Download or Data Streaming services. In the sample workspace attached (ReceivePost.fmw), we simply use a Text File writer to return the same data.

Note: The sample workspace, ReceivePost.fmw, in the “Sample” folder may be used instead of creating your own workspace.

4. Open the ReceivePost.fmw workspace. From the Navigator Panel, select User Parameters and right click to "Manage Parameters". 



Ensure that the Required checkbox is disabled for the Source Text File(s). In order for the call to work, the source file needs to be optional (not required). The Destination Text File can remain enabled for the Required checkbox. 
 

Create Folders in FME Server & Link to FME Workbench

5. Log into your FME Server web interface using your admin account. Your admin account should be enabled with sufficient permissions for the following steps. The default permissions for the fmeauthor role will work too, but for best practice, use your admin account here. Go to the Files & Connections tab and click Resources. On the Resources page, select the Data folder and click +New. In the New Folder Name dialog box, enter "SendSourceData". This will be the folder that the workspace will be writing out to. 





6. Select the newly created folder "SendSourceData", click "Actions" and go to "Properties".



A new window will appear, giving the folder's file properties. Copy the SystemPath: "$(FME_SHAREDRESOURCE_DATA)/SendSourceData".



7. Go back to FME Workbench. In the Navigator panel, under User Parameter, right click "[DestDataset_TEXTLINE] Destination Text File:" to edit Value. Paste the copied System Path: <$(FME_SHAREDRESOURCE_DATA)/SendSourceData> into the Destination Text File value.

Append "/out.txt" to the end of the pasted path with the final Destination Text File value as 
<$(FME_SHAREDRESOURCE_DATA)/SendSourceData/out.txt>.



Save the workspace.
 

Publish the Workspace to FME Server

8. In FME Workbench, click the "Publish to FME Server" button or select File > Publish to FME Server from the menu bar. Connect to your FME Server and publish the workspace into a new repository named “training”.




 

9. Ensure "Upload data files" is enabled and click "Select Files..." to ensure the path to your Sample.txt file is correct. Select OK and click Next.



10. In the last dialog box of the Publish to FME Server wizard, click on the Edit button beside the Data Download service.

First, ensure that the "Send HTTP Message Body to Reader" parameter is set to your reader. You will see another parameter called "Include Writers in Download". Ensure this is set to your Text File Writer.

11. Repeat step 10 for the Data Streaming and the Job Submitter services. Ensure Data Download, Data Streaming, and Job Submitter services are selected and click Publish. 

Create a Webhook

12. On the FME Server web interface, go to Run Workspace and select the Repository "training" and the Workspace "ReceivePost.fmw" that was just published. Select the Job Submitter service and ensure that the Destination Text File ends with ".../out.txt". 



Click Run to ensure the Workspace runs successfully.

13. Once the ReceivePost.fmw is complete and translation is successful, go to Files & Connections > Resources > Data > SendSourceData > out.txt and delete this file. To delete the file, select the file and go to Actions and click Remove. We will be running the workspace via REST API next. 

14. Return back to the Run Workspace page. Open the Advanced panel and select "Create a Webhook". The webhook allows a third-party software to programmatically run this workspace. Note that the API token associated with this webhook allows access to run any other workspaces in the same repository that are registered to the same service. 



15. On the Create Webhook page, a Webhook URL for that repository will be automatically filled with the Token Name, Description, and Expiration date. You may choose to change the expiration date here. Enable "User can upload", and select OK. We will manage the Token's permissions after the webhook is created. 



16. Download webhook to save the information. This is the only time you will be able to access the token and instructions to use this webhook, so don't skip this step!

17. Now that the webhook is created, go to the FME Server User Settings (top right corner) and select Manage Tokens.



This will bring you to the API Tokens page. Click on the "Webhook-ReceivePost from training" token that was created and let's update the permissions.

18. In the Token's Permissions panel, enable Access to Repositories, and expand with the down arrow to reveal options to enable Download, Read and Run for the "training" repository. Next, enable Create for Resources, and expand with the down arrow to enable Access, List, Write, and Upload for the Data Folder. Then, select Advanced for Run Workspace. Finally, click the down arrow for Services to select the type of services you need with this webhook Token. Enable Full Access to Data Download, Data Streaming, Data Upload and Job Submitter if you want to only use one webhook for these services. Click OK to accept the new permissions to your webhook token. 

Using Postman to Make an HTTP POST Request

19. Download Postman using this link https://www.postman.com/downloads/. Postman is a REST client Tool to make calls to FME Server. 

20. Open Postman and select "Skip signing in and take me straight to the app". 



In the Create New dialog box, click on "Request" to create a basic request.



21. Fill in the request and click Save. Next, we receive an option to save our request. This is used if you would like to save the call to use it in the future. Name the call "MyFirstCall" and create a new collection for this article. To create a new "SendSourceData" collection, click +Create Collection. Name and save your collection. 



22. Before posting to FME Server, let's start with a GET request to do a health check from FME Server using the REST API. 

Use the GET method and the following URL to send the request to your FME Server. Please note, you will need to replace <yourServerHost> with your FME Server hostname.

http://<yourServerHost>/fmerest/v3/healthcheck?textResponse=false

The Body should return a JSON response of "status" : "ok" (200), which is a standard response for successful HTTP requests. Note: This will be the only call that does not require a token. 



23. Now let's proceed with the HTTP POST request. In Postman, create a new request by clicking the + on the tab. Change the HTTP request from GET to POST.

From the downloaded webhook file "fmewebhook-Webhook-ReceivePost from training -<yourToken>.txt", copy the Authorization with Header URL and paste into the "Enter request URL" parameter in Postman. 

http://<yourServerHost>/fmejobsubmitter/training/ReceivePost.fmw?SourceDataset_TEXTLINE=%24(FME_MF_DIR)Sample.txt&DestDataset_TEXTLINE=%24(FME_SHAREDRESOURCE_DATA)%2FSendSourceData%2Fout.txt&opt_showresult=false&opt_servicemode=sync

Go to the Headers tab and add a new header. For the Header Key, enter "Authorization", and for the Header Value, enter "fmetoken token=<yourToken>".



You may save this Authorization Header Key as a preset by selecting Presets and Manage Presets. Click Add and name the Header Preset: "Token SSD" and fill in the key, value, and description (expiry date). 



24. Click Send in Postman and a Status 200 OK should return, signifying a successful HTTP request.




25. Since our token also has permission to the Data Download and Data Streaming services, we can simply change the request URL from "fmejobsubmitter" to "fmedatadownload".

In Postman, create a new request by clicking the + on the tab. Ensure the HTTP request is set to POST. From the downloaded webhook file "fmewebhook-Webhook-ReceivePost from training -<yourToken>.txt", copy the Authorization with Header URL and paste into the Enter request URL parameter in Postman. Change the fmejobsubmitter to fmedatadownload.


http://<yourServerHost>/fmedatadownload/training/ReceivePost.fmw?SourceDataset_TEXTLINE=%24(FME_MF_DIR)Sample.txt&DestDataset_TEXTLINE=%24(FME_SHAREDRESOURCE_DATA)%2FSendSourceData%2Fout.txt&opt_showresult=false&opt_servicemode=sync

In the Headers tab, click Presets and select the Token SSD to load in the Authorization key that was previously saved. Click Send.

A Status 200 OK should return. Under Body, select "Preview" to format the HTML. You will find the link to download the zip file. Paste the zip file link to your web browser and the data will be downloaded onto your machine. 



26. Repeat step 25 for Data Streaming service; except switch out the "fmedatadownload" for fmedatastreming" in the URL. Once you click Send, a Status 200 OK should return. In the return Body, the Data Source will be visible in the text format that was uploaded from the Sample.txt file.

http://<yourServerHost>/fmedatastreaming/training/ReceivePost.fmw?SourceDataset_TEXTLINE=%24(FME_MF_DIR)Sample.txt&DestDataset_TEXTLINE=%24(FME_SHAREDRESOURCE_DATA)%2FSendSourceData%2Fout.txt&opt_showresult=false&opt_servicemode=sync



To upload a text file as your source data, return back to the Headers section. Under the Authorization Key, we will be adding two more Keys. First, we will add Content-Type with the Value set to text/plain. Next, we will add the Accept Key, with the Value set as application/json. 



In the Body tab, select binary and upload the test.txt file from the Sample folder. You may choose to create your own text file or edit the test.txt file. Click "Send" and the Status should return 200 OK with the return Body "Hello this is sending source to FME Server with HTTP POST".



Congratulations! Now you’ve successfully posted data to FME Server and triggered a workspace which reads this data as a source dataset using several different FME services!

Please refer to the FME Server 2019 REST API Training for more on working with FME Server REST API and Postman.
 

Additional Resources

Tutorial: Getting Started with APIs

Getting Started with FME and the Web

How to Connect to Any REST API (Without Writing Any Code)

How to Build Complex Calls to APIs (Without Writing Any Code)

Monitoring FME Server Job Activity using the REST API