Tutorial: Creating Web Services using FME Server

Introduction

This article shows you how to use FME Server to drive the operations of a REST API. Here, we’ll look at common scenarios for integrating FME Server with an API gateway, and go through a more specific example.

Requirements

• Web-accessible FME Server 2017.1+ installation (FME Cloud instance works)
• Amazon AWS account
• Basic familiarity with FME Desktop and Server
• Basic familiarity with use of AWS

Introduction to REST Web Services

Web services allow you to interact with a server or another node on a web network using the simplicity of the HTTP standard. The REST concept itself is not a standard, but rather a set of guidelines that promote simple, easy-to-use web APIs.

The guidelines include using:

• HTTP URLs to identify resources (e.g. http://example.com/software/fme)
• HTTP Request Methods (DELETE, GET, POST, PUT) to perform actions on resources.

The majority of REST APIs will accept and respond with specific variants of either JSON or XML formatted data. In the last few years, we have seen a movement away from XML towards JSON as it is more easily supported in web applications. A web service built on FME Server could support XML or JSON, but we’ll use JSON in this tutorial.

SOA, Microservices and APIs

It is becoming increasingly common to design applications and systems as a collection of services, rather than a monolithic whole. Depending on the scale of implementation, you may hear terms like service oriented architecture (SOA) or microservice architecture. While there are many details open to debate, the core idea is that there should be clearly-defined boundaries between components of a system, and they should communicate using well-defined protocols. This results in a collection of services that carry out smaller tasks, rather than a single application that fulfills all requirements. All these services may be implemented using different technologies, and are likely managed by different teams.

However, it is often still desirable to provide a unified interface (API) that synthesizes the data and operations from the different services. This is done using an API gateway - essentially another service that provides the unified interface, and communicates with all the disparate services in the background.

Borrowing from this concept, we will be using FME Server to provide a workspace-based service. We will then build a REST API using AWS API Gateway, and link it to the FME Server web service. In future iterations of this system, additional workspaces, FME Server instances, or even web services provided by entirely different products, could be added for additional functionality.

Alternatives to AWS API Gateway

We have chosen to use AWS API Gateway in this tutorial because we already use AWS for FME Cloud and other solutions. However, there are several other options, both cloud based and self-hosted (on-premises).

If you want to use a different cloud service provider, you could look into solutions such as Apigee Edge or 3scale. If you're looking for an on-premises solution, you can consider products such as Nginx API Gateway or Kong. (This is by no means an exhaustive list).

Scenario

In this tutorial, a city is starting to roll out an API for public and internal use. The Parks Board is the first department to build a web service, but other departments will be doing the same soon. The Parks Board has decided to use FME Server to implement the service, but this will actually be invisible to users of the API. Other departments may choose to use FME Server, other software providing GIS web services, or simply host some static files if their data is fairly simple and not updated frequently. This will all integrate neatly into the same API.

Required functionality

Users need to be able to carry out the following using the parks service:

• Retrieve all of the parks
• Retrieve a specific park
• Search parks by name
• Add a new park
• Update a park
• Delete a park

The operations should properly be restricted to specific users. Querying the service is available to members of the public, but only internal users are permitted to modify the data.

The operations should be accessible through a RESTful web service, which can be expanded in the future to include services from other municipal departments.

Implementation

Designing the API

Each of the required actions on a resource needs to be mapped to an HTTP action on a URL endpoint in the API:

Technical decisions

We’ll be using FME Server to build the parks web service, since FME makes it easy to properly format the data from the Parks Board internal database. However, the job-based workflow of FME Server isn’t ideally suited to building a REST API. Instead, we are going to use a specialized API Gateway. Specifically, we’ll be using aptly-named Amazon AWS API Gateway.

Since we are distributing GIS data through the web, a natural choice for request and response types is GeoJSON.

Setting up a simple API call, end-to-end

It’s good practice to first implement a simple example end-to-end to demonstrate that your systems are communicating properly. In this case, we will implement the /parks query, which just involves a translation from MapInfo TAB to GeoJSON. This will require configuring data streaming, security, and the API Gateway.

Authoring the workspace

The workspace is very simple for this first API call. We’ll just be using the Parks.tab file from the FME sample data package. For your convenience, there is a workspace template available.

Download workspace template

Save the workspace as getAllParksGeoJSON.fmw

Publishing the workspace

Next, we publish the workspace to FME Server. In this tutorial, we’ll be publishing to an FME Cloud instance, since it can be easily accessed from AWS API Gateway.

1. Start publishing the workspace, and log in to the FME Cloud instance.
2. Create a new repository called parksapi
3. Make sure the option to to upload the source data is enabled
4. Associate the workspace with the Data Streaming service. Then, edit its properties, and select the GeoJSON writer as the streaming data source:
   

   

Creating an API user

Now, let’s create an API user. Note that this user should have all privileges required to make calls to the API. The actual authorization of external vs. internal users is done at the gateway level.

Steps to set up the API user:

1. Go to Manage > Administration > Security
2. Create a user, parksapi
3. Give the user the following permissions:
   • Repositories > parksapi > Read, Run
   • Run Workspace > Access
4. Apply the changes

Now, let’s generate a token for the user:

1. Visit https://my-fme-server-example.fmecloud.com/fmetoken
2. Enter the username and password for the API user
3. Save the token for later use

The Web Service URL

We need to get the URL to run the workspace via the Data Streaming service:

1. Go to Run Workspace.
2. Select the parksapi repository, and then select the workspace you published. The only option should be for Data Streaming.
3. Open the Advanced panel
4. Save the base URL & path of the Direct Url Example (make sure to not include any query parameters). For example, https://my-fme-server-example.fmecloud.com/fmedatastreaming/parksapi/getParksGeoJSON.fmw.

Integrating with AWS API Gateway

Log in to your AWS Management Console, and go to Networking & Content Delivery > API Gateway. Now, we’ll create a new API, and add a method to it.

1. Click on the Create API button to create a new API
2. Name the API Municipality API
3. Select Create Resource in the Actions menu of the new API
4. Create a resource called Parks. The default URL, “parks,” is fine.
5. Next, create a method for the resource. This will show up as an empty dropdown menu in the left pane:
   

   

   Select GET as the action.

6. Set up the request. Set the Integration type to HTTP, and use the URL from above as the HTTP endpoint:

   

   After saving the request, you should see a diagram with four main parts - select Integration Request
7. Select Add Query String under “URL Query String Parameters”
8. Name the query string parameter token and enter stageVariables.token. This will allow you to just set the token once for the entire API once you’ve deployed it.
9. Back at the method diagram, click the Test icon in the top left corner. You’ll need to enter the token you generated earlier. Testing the method should result in a response similar to the following:
   

   

Deploying the API

Now that we’ve created an API with one resource and one method, we can deploy it and try it out.

1. Click Actions > Deploy API, and select New Stage for the stage
2. Create a new stage called dev
3. On the dev Stage Editor page, go to the Stage variables tab.
4. Add a stage variable called token and give it the token value you created earlier.
   

   

Testing the API

1. Still on the Stage Editor page, copy the “Invoke URL” for the API, and open a new browser tab
2. Paste the URL, add /parks to the end of it, and press enter. (The full URL will look something like https://0aa000aaa0.execute-api.us-west-2.amazonaws.com/dev/parks).
3. You should now get the example response from the API.

Note that future methods you create can also use the token you specified in the stage variable, so if the token changes, you will be able to just set at the stage level. You could do something similar for the server URL if you have separate development and production FME Servers.

Next Steps

Using this example, we can implement further API calls using FME Server workspaces. Some ideas to work with:

• Published parameters are easily set using query string parameters
• POST data can be read using a Text File reader, or a GeoJSON reader if the data is of that format
• API gateways can often do sophisticated mapping of data. For example, you could use the id of parks to add a link to the URL for that park's data in the summary /parks call. You would want to avoid adding this logic to a workspace, as it requires the web service to have awareness of the API layer.