Send Source Data to FME Flow with HTTP POST

Introduction

To support real-time data applications, you may want to POST data to FME Flow and trigger a workspace that reads this data as a source dataset. For example, you can create triggers for sensors that can send sensor data to an FME Workspace for processing, or you could create your own application that submits form data to FME Flow.

Please Note:

• Using the method described in this article, FME Flow accepts raw data in the POST body only, but does not accept form-data. Form-data can be used to submit workspace parameters simply by sending parameter names and values as form element names and values.
• Currently, this method is best for simple text data such as XML or JSON and does not support multipart file upload. If you want to upload file data to be used as the source dataset for an FME Workspace, please see the article Data Upload with FME Flow. 

The following Web Services can accept raw data sent to FME Flow using HTTP POST:

• FME Data Download
• FME Data Streaming
• FME Job Submitter

You will be using FME Flow REST API and creating your own application to solve data challenges and ultimately to leverage the power of FME. A REST API allows a user to interact with the server by creating requests and receiving responses using HTTP Protocols. Requests are made through URLs, and responses are formatted in JSON. HTTP Methods are essential to every call to the REST API. In other words, HTTP Methods represent an action to FME Flow that you would like to complete. The HTTP Method includes GET, POST, PUT, or DELETE. We will be focusing on the HTTP POST request, which is used to create a publication, project, or notification.

Visit the FME Flow REST API page to learn more about the different actions and requests that are available. You can also access this page via your FME Flow web interface under the Help icon by opening the REST API tab.

Postman is a REST Client Tool to make calls to FME Flow. If there is another REST client tool you are familiar with, you may use that as an alternative. The instructions in this article will refer to the Postman interface.

Webhooks are URLs that allow external software to run the workspaces. To learn more about webhooks and FME Flow, read the Webhooks and FME Flow article.

Requirements

• Download Postman
• Sample.zip attached to this article

Step-by-step Instructions

Prepare the Data and Create a Workspace

Note: You may use the sample workspace, ReceivePost.fmw, in the “Sample” folder instead of creating your own workspace.

1. Create a simple text file that is structured to represent a sample of data that you will be sending (e.g., an XML or JSON snippet). In the “Samples” folder attached to this tutorial, a very simple data file called Sample.txt may be used. This text file will be called later on when posting to FME Flow with the Data Streaming service.

2. Build an FME workspace that will read the data you POST to it. In most cases, using the Text File Reader is best. Even if the data is XML or JSON, you can still use the Text File reader. Make sure to set the reader parameter "Read Whole File at Once" to Yes. You can then parse the data further along in the workspace using the appropriate XML or JSON transformers.

3. Add a writer to the workspace for output data - which will be sent to the Data Download or Data Streaming services. In the sample workspace attached (ReceivePost.fmw), we simply use a Text File writer to return the same data.

4.  From the Navigator Panel, right-click on User Parameters and select "Manage Parameters".

Ensure that the Required checkbox is disabled for the Source Text File(s). In order for the call to work, the source file parameter needs to be optional, so ensure that the Required box is not checked. The Destination Text File can remain enabled for the Required checkbox.

Create Folders in FME Flow & Link to FME Workbench

5. Log into your FME Flow web interface using your admin account. Your admin account should be enabled with sufficient permissions for the following steps. The default permissions for the fmeauthor role will work too, but for best practice, use your admin account here. On the Resources page, select the Data folder and click New. Name the new folder "SendSourceData" and click Create. This will be the folder that the workspace will be writing out to.





6. Select the newly created folder "SendSourceData", click "Actions" and go to "Properties".



A new window will appear, giving the folder's file properties. Copy the SystemPath: 

 "$(FME_SHAREDRESOURCE_DATA)/SendSourceData"

7. Return to FME Workbench. In the Navigator panel, expand User Parameters, right-click the [DestDataset_TEXTLINE] Destination Text File parameter, and select Edit Value. Paste the copied System Path into the Destination Text File value:

 $(FME_SHAREDRESOURCE_DATA)/SendSourceData

Finally, specify the output file name and format by appending "/out.txt" to the end of the System Path:

$(FME_SHAREDRESOURCE_DATA)/SendSourceData/out.txt

Save the workspace.

Publish the Workspace to FME Flow

8. In FME Workbench, click the "Publish to FME Flow" button or select File > Publish to FME Flow from the menu bar. Connect to your FME Flow and publish the workspace into a new repository named “Training”.

Select a valid FME Flow connection, then click "New..." to create a new repository for the workspace. Name this new repository "Training" and click OK.

9. Ensure "Upload data files" is enabled. Click "Select Files..." to ensure the path to your Sample.txt file is correct. Select OK and click Next.

10. In the last dialog box of the Publish to FME Flow wizard, click on the Edit button beside the Data Download service.

First, ensure that the "Send HTTP Message Body to Reader" parameter is set to your reader. Below this is another parameter called "Writers in Download". Ensure this is set to your Text File Writer.

11. Repeat step 10 for the Data Streaming and the Job Submitter services. Ensure Data Download, Data Streaming, and Job Submitter services are selected and click Publish.

Create a Webhook

12. On the FME Flow web interface, go to Run Workspace. Select the Repository "Training" and the Workspace "ReceivePost.fmw" that was just published. Select the Job Submitter service and ensure that the Destination Text File ends with ".../out.txt".

Click Run to ensure the Workspace runs successfully.


13. Once the ReceivePost.fmw is complete and the translation is successful, go to Resources > Data > SendSourceData > out.txt and delete this file. To delete the file, select it, go to Actions, and click Remove. Next, we will run the workspace via REST API.

14. Return to the Run Workspace page. Open the Advanced panel and select "Create a Webhook". The webhook allows third-party software to programmatically run this workspace.

Note: The API token associated with this webhook allows access to run any other workspaces in the same repository that are registered to the same service.



15. On the Create Webhook page, a Webhook URL for that repository will be automatically filled with the Token Name, Description, and Expiration date. You may choose to change the expiration date here. Enable "User can upload", and select Create. We will manage the Token's permissions after the webhook is created.



16. Download webhook to save the information. This is the only time you will be able to access the token and instructions to use this webhook, so don't skip this step!

17. Now that the webhook is created, go to the FME Flow User Settings (top right corner) and select Manage Tokens.


This will bring you to the API Tokens page. To update the permissions, click on the "Webhook-ReceivePost from training" token that was created.

18. In the token's Permissions panel, enable Access to Repositories, and expand with the down arrow to reveal additional options. Enable Download, Read, and Run for the "training" repository.

Next, enable Create for Resources, and expand with the down arrow to enable Access, List, Write, and Upload for the Data Folder.

Select Advanced for Run Workspace.


Finally, click the down arrow for Services to select the type of services you need with this webhook token. Enable Full Access to Data Download, Data Streaming, Data Upload, and Job Submitter if you want to only use one webhook for these services. At the bottom of the page, click Save to apply the new permissions to your webhook token.

Using Postman to Make an HTTP POST Request

19. Download Postman using this link https://www.postman.com/downloads/. Postman is a REST client tool to make calls to FME Flow.

20. Open Postman and select "Continue without an account".



From the main Postman workbench view, click the "+" icon to create a new HTTP request.



21. Before posting to FME Flow, let's start with a GET request to do a health check from FME Flow using the REST API.

Use the GET method and the following URL to send the request to your FME Flow. Please note, you will need to replace <yourServerHost> with your FME Flow hostname.

http://<yourServerHost>/fmerest/v3/healthcheck?textResponse=false

Click Send. The Body should return a JSON response of "status" : "ok" (200), which is a standard response for successful HTTP requests.

Note: This will be the only call that does not require a token.

22. Let's proceed with the HTTP POST request. In Postman, create a new request by clicking the + on the tab. Change the HTTP request from GET to POST.

From the webhook file that you downloaded in Step 16 (fmewebhook-Webhook-ReceivePost from training -<yourToken>.txt), copy the Authorization with Header URL and paste into the "Enter request URL" parameter in Postman.

http://<yourServerHost>/fmejobsubmitter/training/ReceivePost.fmw?SourceDataset_TEXTLINE=%24(FME_MF_DIR)Sample.txt&DestDataset_TEXTLINE=%24(FME_SHAREDRESOURCE_DATA)%2FSendSourceData%2Fout.txt&opt_showresult=false&opt_servicemode=sync

Go to the Headers tab and add a new header. For the Header Key, enter "Authorization", and for the Header Value, enter "fmetoken token=<yourToken>".

23. Click Send in Postman and a Status 200 OK should return, signifying a successful HTTP request.

24. Since our token also has permissions enabled for the Data Download and Data Streaming services, we can access those services by simply changing a string in the request URL from "fmejobsubmitter" to "fmedatadownload".

In Postman, create a new request by clicking the + on the tab. Ensure the HTTP request is set to POST. Copy the Authorization with Header URL from your downloaded webhook file into the field. This time, we will edit the URL: change the "fmejobsubmitter" string to "fmedatadownload".

http://<yourServerHost>/fmedatadownload/training/ReceivePost.fmw?SourceDataset_TEXTLINE=%24(FME_MF_DIR)Sample.txt&DestDataset_TEXTLINE=%24(FME_SHAREDRESOURCE_DATA)%2FSendSourceData%2Fout.txt&opt_showresult=false&opt_servicemode=sync

Navigate to the Headers tab and enter the same values that were used previously: for the Header Key, enter "Authorization", and for the Header Value, enter "fmetoken token=<yourToken>". Click Send.

A Status 200 OK should return. Under Body, select "Preview" to format the HTML. You will find the link to download the zip file. Paste the zip file link to your web browser and the data will be downloaded onto your machine.

25. Repeat step 25 for the Data Streaming service, except this time, replace the "fmedatadownload" string in the URL with "fmedatastreaming."

http://<yourServerHost>/fmedatastreaming/training/ReceivePost.fmw?SourceDataset_TEXTLINE=%24(FME_MF_DIR)Sample.txt&DestDataset_TEXTLINE=%24(FME_SHAREDRESOURCE_DATA)%2FSendSourceData%2Fout.txt&opt_showresult=false&opt_servicemode=sync

Once you click Send, a Status 200 OK should return. In the return Body, the Data Source will be visible in the text format that was uploaded from the Sample.txt file.

To upload a text file as your source data, return back to the Headers section. Under the Authorization Key, we will be adding two more Keys-Value pairs:

• Key: "Content-Type", Value: "text/plain"
• Key: "Accept Key", Value: "application/json"
  
  

In the Body tab, select binary and upload the test.txt file from the Sample folder. You may choose to create your own text file or edit the test.txt file. Click "Send" and the Status should return 200 OK with the return Body "Hello this is sending source to FME Flow with HTTP POST".

Congratulations! You’ve successfully posted data to FME Flow and triggered a workspace that reads this data as a source dataset using several different FME services. 

Additional Resources

Tutorial: Getting Started with APIs

Working with FME and Web Data, Services, and Protocols

How to Connect to Any REST API With no Code

How to Build Complex Calls to APIs (Without Writing Any Code)

Monitoring FME Flow Job Activity using the REST API