Creating Web Service Connections in FME Using the OAuth 2.0 Protocol

Safe Software Support Team Member
Safe Software Support Team Member
  • Updated

FME Version

  • FME 2022.1

Introduction

This article describes how to set up a new web service and web service connection using the OAuth 2.0 authorization protocol. Safe Software has already registered applications with many commonly used web services, providing convenient out-of-the-box web service options to users. These out-of-the-box services make use of credentials (client ID and client secret) held by Safe Software and will work well in a testing environment. However, in production environments, it is recommended that you create a new OAuth 2.0 web service and connection using your organizational or personal credentials. In this way, you can directly handle web service outages or other troubleshooting tasks, since the credentials would be tied to you or your organization rather than to Safe Software. 

This article assumes prior knowledge of APIs and familiarity with the OAuth 2.0 protocol. For more information on APIs and FME, please see the Getting Started with APIs tutorial. Please see the OAuth section of this article for an overview of OAuth 2.0 in the context of FME web connections, as well as some key terminology.

 

Step-by-Step Instructions

The following instructions will walk you through the creation of a user-defined OAuth 2.0 web service connection, beginning outside of the FME platform.

1.  Create an Account for the Web Service of Interest
First, make sure you have a user account established for the web service to which you would like to connect. Generally, a basic user account will suffice for the purposes of creating an FME web service connection. There are, however, some web services that require a developer account. It is best practice to first read through the web service’s documentation to determine what account level you will require.

2.  Register a New Application with the Web Service  
Next, you must register a new application (sometimes called a new project) with the web service’s authorization server. This is where you are registering FME as a client application of the web service, letting the web service know that FME can act on your behalf in accessing and retrieving resources from the web service API. Check the developer or API section of the web service’s documentation for information on OAuth 2.0 application registration and for a link to where you can create a new application or project with the web service.

Registering a new application or project with a web service may require the input of the following parameters:

  • Application name (often in the form of <Company Name><Service Name>);
  • An application icon;
  • Application description;
  • Application home page URL;
  • Type of application (to determine if a client_secret must be issued);
  • Redirect URI (aka Callback URL; typically not optional).


Other parameters may be required, as there is no set standard for application registration. The only parameter which is typically required for registration is the Redirect URI. This is a URL location where the authorization server will redirect the user once the application has been authorized (or denied) and given an access token. Careful attention must be paid to this parameter, as many OAuth 2.0 errors are caused by incorrectly formatted redirect URIs. Consult the web service documentation for instructions on formatting the redirect URI.

Note: For FME Workbench, set the Application Name to Safe Software FME, and set the home page URL to safe.com. If the web service does not provide a redirect URI, you can use http://localhost for FME Desktop, and https://<public FME Server Address>/fmeoauth for FME Server. Keep in mind these parameters may or may not apply to the particular web service you intend to connect to.


3.  Obtain Client ID and Client Secret
Once the new application has been registered with the authorization server, the server will often respond by assigning a client ID and client secret to the client application. These strings are similar to a username and password, and are used to generate an access token from the authorization server. Client ID and client secret are often automatically generated upon registration, but some web services do have additional steps required to obtain these strings. Consult the API documentation for the web service you wish to connect to for more information on generating client ID and client secret.

It is important to note that many services do not provide a client secret. When a client secret is not issued, FME Desktop can accommodate by allowing client secret to be an optional parameter of the web service definition, as described in the Client Information portion of Step 5. Sending a client secret when the service doesn’t expect it, or not sending client secret when it is expected, are both common causes of errors when attempting to authorize a web service.

Be sure to securely store the client ID and client secret assigned to your application, as many web services will only display this information once at registration. You will need these values to set-up the OAuth 2.0 web service definition in FME Desktop. Additionally, while registering your application with some web services, you will be asked to specify your application type. Selecting the type of application you are registering will often determine if a client secret will be issued. Choose Web Application if you are registering FME Server, and Desktop Application for registering FME Desktop. If the generation of client ID and client secret requires additional steps beyond initial application registration, be sure that you specify the same Application Name and Redirect URI you used for registration.     

4.  Open a Web Service Definition Template in FME Workbench
Open FME Workbench and navigate to the Tools menu. Select FME Options, and then select Web Connections from the left-side menu.  Next, select Manage Services at the bottom right of the window
 
ManageServices.jpg

The Manage Web Services window will display a list of the current web services available in FME. You can see the definition of each web service by highlighting it in the list. The definition will display on the right side of the window.

ManageWebServicesOrientation.jpg

This list contains out-of-the-box as well as user-defined web services, with no obvious way to distinguish between them other than the Web Service Name and Connection Description parameters. It is important, therefore, to thoughtfully name and describe your user-defined web services upon creation so that you can easily distinguish them from the out-of-the-box web services included with FME.

Before initiating a new, blank web service definition template, it is recommended that you first examine the FME out-of-the-box web service definition for your service of interest (if available). Select your web service of interest from the web services list in the Manage Web Services dialog. Some of these definitions are meant to serve as pre-formatted templates to allow for easier user definition of the more complicated web services. These pre-formatted templates can be quickly adjusted by following the directions in the Web Service Setup Instructions section of the service definition.

SharepointTemplateResized.jpg

From the Manage Web Services window, select the “+” symbol below the list of web services. A drop-down menu of web service formats will display. Select OAuth 2.0 Service to open a new template for creating a new web service which uses the OAuth 2.0 authorization protocol.

Note: if you have decided to use one of the pre-formatted web service definitions as a template, select Create From instead of OAuth 2.0 Service, and then choose your web service definition of interest from the list that displays. Follow the instructions in the pre-formatted template, if provided. If instructions are not provided in the pre-formatted template, follow the instructions below.

ManageWebServicesPlusOAuthCombine.jpg

5.  Populate the Web Service Definition Template
This step will go through filling out the New Web Service template, section by section. These instructions can also be used to help adjust the parameter values in the pre-formatted web service definition templates, as needed.
 

The Overview Section

In the New Web Service window that opens, enter a meaningful Web Service Name and Connection Description for your web service. Again, make sure you can use these parameters to distinguish this user-defined web service from the other web services on the list at left. If you are using a pre-formatted template, make sure the instructions therein (if provided) do not prohibit web service name changes.
 
OverviewSection.jpg


Client Information Section

Next, fill out the Client Information section with the client ID and client secret that was generated previously. If your app was not issued a client secret, check the box next to Optional, at the end of the Client Secret text box.  Also, enter the same Redirect URI used earlier at application registration.

ClientInformationSection.jpg


The Authorization Parameters Section

The first step in obtaining access to a web service via OAuth 2.0 is to get authorization from the user. In FME, this is accomplished by displaying an interface to the user that is provided by the web service. The Authorization Parameters section should contain the URL which points to this interface. This URL is typically found in the developer documentation for the web service, most often in a section titled “Authentication” or “OAuth 2.0”. If these titles aren’t present, check in the “Getting Started” section, or use a document search feature (ctrl + F, for example) to perform a keyword search for relevant information.

AuthorizationParametersSection.jpg

Note: occasionally, the Authorization URL will require specific parameters appended at its end to limit the scope (or permissions) or to specify the response type of the authorization request. The Google suite of web services, for example, has these additional authorization URL requirements. Check the web service documentation for these additional parameters and how to format them correctly.
 

Retrieve and Refresh Token Parameters Sections

The Retrieve Token Parameters section of the template is where you configure access token acquisition from the web service API. To do so, you will need to know the web service API endpoint you wish to access. This endpoint URL can typically be found within the web service API documentation under “Obtaining an Access Token” or similar. Once this parameter is properly configured, an access token will be generated upon user authorization. This token is then used by FME to make secure API calls on behalf of the user, limited to the scopes specified by the user, until the token expires or is revoked. Essentially, the access token lets the web service API know that FME has the user’s permission to access certain protected resources and perform certain authorized tasks.

RetrieveRefreshPreview.jpg

The Refresh Token Parameters section controls how a new, valid access token is generated from the web service API once the original access token has expired or is revoked. It is therefore important, particularly if publishing the workflow to FME Server, to configure this parameter correctly so that a valid access token can be generated as needed. So long as the refresh token remains valid, new access tokens can be acquired. The Request Format parameter, just below the URL, allows you to specify the desired request format. Typically, the default values are appropriate. This field can also be left blank. 

Sometimes the Refresh Token URL is the same as the Retrieve Token URL, while other times the web service has a dedicated refresh workflow. If there is a dedicated refresh workflow, then a refresh token will have been issued when the original access token was issued. It is best practice to consult the web service API documentation on refresh tokens to get further instructions. 


6.  Test Your OAuth 2.0 Web Service Definition from the Web Service Definition Template    
Now you are ready to test your newly configured web service definition from within the New Web Service template that you have just completed. At the bottom of the New Web Service template, select Apply to save the web service definition. Now, just above the Apply button and below the API Call Parameters, select Test

ManageWebServicesTest.jpg

FME will attempt to connect to the web service using the configurations you just specified and will display the progress of this attempt in the translation log window. If this window is hidden, enable it in FME Workbench under View > Windows > Translation Log. Once you have verified your connection and corrected for errors, you are ready to test using the HTTPCaller transformer in a workspace. Select Close at the bottom of the Manage Web Services window and OK in the FME Options window. For troubleshooting help with Test errors, see Troubleshooting the Web Service Definition Test section below.  
 

7.  Test a New Web Service Connection Using Your Web Service Definition and the HTTPCaller
     Transformer

Navigate to a blank FME Workbench canvas. Add a Creator transformer, and then attach an HTTPCaller transformer to the Creator. Now right-click on the HTTPCaller and select Connect Loggers. You should see two new transformers appear attached to the HTTPCaller. These Logger transformers will log the results of the API calls made by the HTTPCaller.

testworkspace.jpg

Make sure you know the endpoint of the web service API you wish to test, and an idea of the return expected from the connection test. Select the gear icon of the HTTPCaller to open its parameters. Set the HTTP method to the method required for the connection test (GET, POST, etc). Set the Request URL to the desired endpoint of the API.

Now, at the bottom of the HTTPCaller parameters, enable Use Authentication. Select Web Connection for the Authentication Method. In the Web Connection parameter, select Add Web Connection from the drop-down menu.

httpcaller_webconnection.jpg

In the Edit Web Connection dialog that opens, set the Web Service parameter to your newly created OAuth 2.0 web service by finding its name in the drop-down list. Set the Connection Name to a name of your choosing. Select Authenticate at the bottom-right to create the web connection, and sign into your web service user account in the window that opens. This will authorize FME to act on your behalf in accessing and retrieving resources from the web service.  

WebConnectionSave.jpg

Finally, run your workspace to test this new web service connection as part of a translation. Check the translation log to make sure there are no errors. Inspect the output of the HTTPCaller to verify the results are as expected. Once you have verified your web service connection, save your workspace. This workspace will serve as a Test Workspace if you choose to follow through with the next optional step.
 

8.  Share Your Web Connection on FME Hub (optional, requires a free FME Community user account)
Many members of the FME Community choose to share their web service definitions in the FME Hub. The FME Hub is a great place to find new custom transformers, workspace templates, web connections, and other creative projects that FME users have shared with the FME Community. More information on the FME Hub can be found in this article.

If you would like to contribute your OAuth 2.0 web service definition, first login to the FME Community and access the FME Hub from the More tab of the Community homepage. Search the FME Hub to make sure a web service definition does not already exist for the service you would like to share. You can apply a Web Connections filter to your keyword search to make the results easier to parse.

HubSearch.jpg

Once you have verified that your web service definition does not already exist, return to FME Workbench. Navigate to Tools > FME Options > Web Connections. Select Manage Services, at the bottom-right.

ManageServices.jpg

Find your newly created web service definition in the web service list, along the left side of the Manage Web Services dialog. Once selected, scroll to the bottom of the web service definition and, next to the Test button, select Export

ExportConnection.jpg

In the dialog that opens, set the Export to Folder parameter to a location where you would like to save the exported XML definition of the web service. Disable the Include Credentials option, and select OK. Disabling the Include Credentials option will require other FME users to provide their own web service credentials for authorization rather than using your credentials.

ExportParameters.jpg

Now, with your XML export and your Test Workspace (from Step 7) in hand, return to FME Hub in your browser. Follow the instructions outlined in this article to complete the web service definition upload to FME Hub.
 

Using the FME Hub OAuth 2.0 Web Service Definitions

Another option for populating a web service definition for a service of interest is to download one of the many formatted OAuth 2.0 web service definitions available from FME Hub. Performing an FME Hub search for "OAuth 2.0 web services", filtered by Web Connections (as you did in Step 8 above), will return several available OAuth 2.0 web service definitions. Select the definition of interest, and then select Download on the page that opens. You will receive an XML document containing the necessary specifications of the web service definition.

DownloadFromHub.jpg

Return to FME Workbench, and access the Manage Web Services dialog from the Web Connections menu of FME Options (Tools > FME Options > Web Connections > Manage Services button at the bottom-right). Below the list of web services therein, click on the "+" symbol (as you did in Step 4 above ), and choose Import From File. In the dialog that opens, click on the ellipses next to the File to Import text box. Navigate to your downloaded XML web service definition, and click Open. Unless the FME Hub description of the web service explicitly states a password, you can ignore the Password parameter and click OK. 

ImportXML.jpg

Your newly uploaded web service definition should open in a pre-formatted definition template. Follow the instructions given in the FME Hub description of the web service to adjust the template. You will likely have to enter your own Client Information parameters, generated by registering your app with the web service. You can save and test this web service definition using the same methods discussed in Step 6 and Step 7, above. 
 

Troubleshooting the Web Service Definition Test  

In Step 6 above, you tested the web service definition from the Manage Web Services dialog. Ideally, the test will allow you to authenticate FME to the web service without error. However, should errors with the test arise, the following troubleshooting options may help resolve the issue:

  • If you receive an error message or code after entering your login credentials in the authentication pop-up, perform a web search for “<Error Code or Message> and <Name of Web Service>”. For example, if you receive a 401 error code from an attempt to authenticate with Google Drive, you could do a web search for 401 error Google Drive”. Often, other users have encountered the same error and have developed ways to fix the error.  
  • Try clearing your FME cache by purging temporary files. This can be done from within FME Workbench by accessing Tools > Purge temporary files.   
  • Try using a web debugging tool, like Fiddler, to analyze the exact response from the web service when authentication fails. Often, a web debugging tool can provide much more detailed information about why the authentication request failed.

 

Additional Resources

How to create an FME Web Connection for Esri ArcGIS Portal Feature Service (OAuth 2.0)
How to create a Sharepoint Web Connection
Setting up a Fitbit Web Service
How to create a web service for Microsoft Graph (Microsoft Teams)  

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.