Tutorial: Creating Web Services Using FME Flow

Introduction

This article shows you how to use FME Flow (formerly FME Server) to drive the operations of a REST API. Here, we’ll look at common scenarios for integrating FME Flow with an API gateway and follow a more specific example.

Requirements

• Web-accessible FME Flow installation (FME Flow Hosted instance works)
• Amazon AWS account
• Basic familiarity with FME Form and Flow
• Basic familiarity with the use of AWS

Introduction to REST Web Services

Web services allow you to interact with a server or another node on a web network using the simplicity of the HTTP standard. The REST concept is not a standard but a set of guidelines promoting simple, easy-to-use web APIs.

The guidelines include using:

• HTTP URLs to identify resources (e.g. http://example.com/software/fme)
• HTTP Request Methods (DELETE, GET, POST, PUT) to perform actions on resources.

The majority of REST APIs will accept and respond with specific variants of either JSON or XML-formatted data. In the last few years, we have seen a movement away from XML towards JSON as it is more easily supported in web applications. A web service built on FME Flow could support XML or JSON, but we’ll use JSON in this tutorial.

SOA, Microservices, and APIs

It is becoming increasingly common to design applications and systems as a collection of services, rather than a monolithic whole. Depending on the scale of implementation, you may hear terms like service-oriented architecture (SOA) or microservice architecture. While there are many details open to debate, the core idea is that there should be clearly defined boundaries between system components, and they should communicate using well-defined protocols. This results in a collection of services that carry out smaller tasks, rather than a single application that fulfills all requirements. All these services may be implemented using different technologies and are likely managed by different teams.

However, providing a unified interface (API) that synthesizes the data and operations from the different services is often desirable. This is done using an API gateway - another service that provides the unified interface, and communicates with all the disparate services in the background.

Based on this concept, we will use FME Flow to provide a workspace-based service. We will then build a REST API using AWS API Gateway and link it to the FME Flow web service. In future iterations of this system, additional workspaces, FME Flow instances, or even web services provided by entirely different products could be added for additional functionality.

Alternatives to AWS API Gateway

We used AWS API Gateway in this tutorial because we already use AWS for FME Flow Hosted. However, several other options are cloud-based and self-hosted (on-premises).

You could look into solutions such as Apigee Edge or 3scale if you want to use a different cloud service provider. You can consider products such as Nginx API Gateway or Kong if you're looking for an on-premises solution. This is by no means an exhaustive list.

Scenario

In this tutorial, a city is starting to roll out an API for public and internal use. The Parks Board is the first department to build a web service, but other departments will do the same soon. The Parks Board has decided to use FME Flow to implement the service, but this will be invisible to API users. Other departments may choose to use FME Flow, another software that provides GIS web services, or host some static files if their data is fairly simple and not updated frequently. This will all integrate neatly into the same API.

Required functionality

Users need to be able to carry out the following using the parks service:

• Retrieve all of the parks
• Retrieve a specific park
• Search parks by name
• Add a new park
• Update a park
• Delete a park

The operations should be properly restricted to specific users. The service is available to members of the public for querying, but only internal users are permitted to modify the data.

The operations should be accessible through a RESTful web service, which can be expanded to include services from other municipal departments.

Implementation

Designing the API

Each of the required actions on a resource needs to be mapped to an HTTP action on a URL endpoint in the API:

Technical decisions

We’ll be using FME Flow to build the park's web service, since FME makes it easy to properly format the data from the Parks Board's internal database. However, the job-based workflow of FME Flow isn’t ideally suited to building a REST API. Instead, we are going to use a specialized API Gateway. Specifically, we’ll be using aptly-named Amazon AWS API Gateway.

Since we are distributing GIS data through the web, GeoJSON is a natural choice for request and response types.

Setting up a simple API call, end-to-end

It’s good practice to implement a simple example end-to-end to demonstrate that your systems communicate properly. In this case, we will implement the /parks query, which involves translating MapInfo TAB to GeoJSON. This will require configuring data streaming, security, and the API Gateway.

Authoring the workspace

The workspace is very simple for this first API call. We’ll just be using the Parks.tab file from the FME sample data package. For your convenience, a workspace template is available.

Download workspace template

Save the workspace as getAllParksGeoJSON.fmw

Publishing the workspace

Next, we publish the workspace to FME Flow. In this tutorial, we’ll publish to an FME Flow Hosted instance since it can be easily accessed from the AWS API Gateway.

1. Start publishing the workspace, and log in to the FME Flow Hosted instance.
2. Create a new repository called parksapi.
3. Make sure the option to upload the source data is enabled.
4. Associate the workspace with the Data Streaming service with the following parameters: 

• Writers in Stream: "Parks[GEOJSON]"

Click OK. 

Creating an API user

Now, let’s create an API user. Note that this user should have all privileges required to make calls to the API. The actual authorization of external vs. internal users is done at the gateway level.

Steps to set up the API user:

1. Go to User Management > Users > Create.
   
   
   
   
2. Fill the required fields: Username (For this tutorial, we will use "parksapi"), Full Name, Password, and Confirm Password.
   
   
3. Give the user the following permissions:
   • Repositories > parksapi > Read, Run
   • Run Workspace > Access
   • Services > Data Streaming > Full Access
     
     
     
4. Click Create to apply the parameters.

Now, let’s generate a token for the user:

1. Log in as the new parksapi user.
2. At the top right of the FME Flow web interface, navigate to User Settings > Manage Tokens.
   
3. Click Create to open the API Token configuration page.
4. Enter a name for the token and select All Permissions.
5. Click Save to generate the token.
6. Click Download Token to save the token details as a .txt file.

The Web Service URL

We need to get the URL to run the workspace via the Data Streaming service:

1. Go to Run Workspace.
2. Select the parksapi repository, and then select the workspace you published. The only option should be for Data Streaming.
3. Open the Advanced panel.
4. Under Other Ways to Run this Workspace, click Create a Webhook to access the direct URL.
5. Save the URL, making sure not to include any query parameters:

   https://<my-fme-flow-example.com>/fmedatastreaming/parksapi/getParksGeoJSON.fmw

Integrating with AWS API Gateway

Log in to your AWS Management Console, and go to Networking & Content Delivery > API Gateway. Now, we’ll create a new API and add a method.

1. Click on the Create API button.
2. Click the Build button for a REST API on the Choose an API Type page.
3. Name the API Municipality API and leave the other parameters on their default settings. Click Create API.
4. Under the Resources heading, select Create resource.
5. Name the resource Parks, then click Create resource.
6. Ensure that the /Parks resource is selected, then click Create method in the Methods pane.
   
7. Apply the following settings in the Method details pane:

• Method type: GET
• Integration type: HTTP
• HTTP proxy integration:On
• HTTP method: GET
• Endpoint URL: <workspace URL> https://<my-fme-flow-example.com>/fmedatastreaming/parksapi/getParksGeoJSON.fmw

Click the Create method to apply the settings. You will be returned to the Municipality API Resources page.

1. Next, navigate to the Integration request tab and click Edit.
2. Scroll down to the URL query string parameters pane and click Add query string parameter.
3. Name the query string parameter token and enter stageVariables.token in the other field. This will allow you to set the token once for the entire API once you’ve deployed it. Click Save.
4. Back at the Municipality API Resources page, navigate to the Test tab. You can leave the Query strings and Headers fields empty, but you must enter the token you generated earlier. Testing the method should result in a response similar to the following:
   

   

Deploying the API

Now that we’ve created an API with one resource and one method, we can deploy it and test it.

1. Click Deploy API at the top right of the page and select New Stage for the stage.
2. Name the new stage dev, then click Deploy.
3. On the Stages page, navigate to the Stage variables tab and click Edit.
4. Click Add stage variable, name it token, and enter the token value you created earlier. Click Save.

Testing the API

1. Still on the Stage Editor page, copy the “Invoke URL” for the API, and open a new browser tab.
2. Paste the URL, add /parks to the end of it, and press Enter. (The full URL will look something like https://0aa000aaa0.execute-api.us-west-2.amazonaws.com/dev/parks).
3. You should now get the example response from the API.

Note that future methods you create can also use the token you specified in the stage variable, so if the token changes, you can just set it at the stage level. You could do something similar for the server URL if you have separate development and production FME flows.

Next Steps

We can implement further API calls using this example of FME Flow workspaces. Some ideas to work with:

• Published parameters are easily set using query string parameters
• POST data can be read using a Text File reader, or a GeoJSON reader if the data is of that format
• API gateways can often do sophisticated mapping of data. For example, you could use the id of parks to add a link to the URL for that park's data in the summary /parks call. You would want to avoid adding this logic to a workspace, as it requires the web service to have awareness of the API layer.