FME Version
Introduction
This article describes how to configure a new FME web service and connection using the OAuth 2.0 authorization protocol. Safe Software has already registered applications with many commonly used web service providers, like ArcGIS Online, providing convenient out-of-the-box web service options to get started. However, these web services use credentials (client ID and client secret) owned by Safe Software and will not work in production environments. Creating a new OAuth 2.0 web service and connection using organizational credentials from your own registered application will allow you to maintain and troubleshoot connection issues, as well as connect to the web service of interest on FME Flow.
The steps in this article are a general guide. Since every web service provider is different, providing steps for each is not feasible. We provide more detailed steps for some common web services linked in the section OAuth 2.0 Web Service Connection Articles. If there is a specific article for the web service you are trying to connect to, please check it first.
This article assumes some prior knowledge of APIs and familiarity with the OAuth 2.0 protocol. For more information on APIs and FME, please see the Getting Started with APIs tutorial. Please see the OAuth section of this article for an overview of OAuth 2.0 in the context of FME web connections and key terminology.
Step-by-step Instructions
Part 1: Register an Application
A user account and application must be registered with the web service provider.
1. Create an Account for the Web Service of Interest
Make sure that you have a user account established for the web service to which you would like to connect. Generally, a basic user account will suffice for creating an FME web service connection. There are, however, some web services that require a developer account. It is best practice to first read through the web service provider’s OAuth 2.0 documentation to determine what account level you need.
2. Register a New Application with the Web Service
Register a new application (sometimes called a new project) with the web service provider’s authorization server. This is where you are registering FME as a client application of the web service provider, letting the web service provider know that FME can act on your behalf in accessing and retrieving resources from the web service provider's API. Check the developer or API section of the web service provider’s documentation for information on OAuth 2.0 application registration.
Registering a new application or project with a web service may require the input of the following parameters:
- Application type;
- Application name (often in the form of <Company Name><Service Name>);
- An application icon;
- Application description;
- Application home page URL;
- Type of application (to determine if a client_secret must be issued);
- Redirect URI (aka Callback URL)
Other parameters may be required, as there is no standard for application registration.
Some web service providers ask you to specify your application type. Choose Desktop Application or an equivalent for registering FME Form.
The Redirect URI is typically required. This is a URL location where the authorization server will redirect the user once the application has been authorized (or denied) and given an access token. Careful attention must be paid to this parameter, as many OAuth 2.0 errors are caused by incorrectly formatted redirect URIs. Consult the web service provider’s documentation for instructions on formatting the redirect URI.
FME Form is a desktop application, and the Redirect URI is typically http://localhost or urn:ietf:wg:oauth:2.0:oob, but it varies depending on the web service.
If the Application home page URL is required by the web service, it can typically be set to any valid URL. For example, safe.com.
3. Obtain Client ID and Client Secret
Once the new application has been registered with the authorization server, it will usually be assigned a client ID and client secret. These strings are similar to a username and password, and are used to generate an access token from the authorization server. Some web service providers have additional steps required to obtain these strings. Make sure to specify the same Application name and Redirect URI you used for registration, if requested. Consult the web service provider’s documentation for specific details.
Some web services may not provide a client secret, depending on the type of application you registered. In this case, the client secret can be set as optional in the web service definition configured in FME, as described in the Client Information portion of Part 2, Step 5. Sending a client secret when the authorization server doesn’t expect it, or not sending a client secret when it is expected, are both common causes of errors when attempting to authenticate a web service in FME.
Be sure to securely store the client ID and client secret assigned to your application, as this information may only be displayed once, at application registration. You will need these values to set up the OAuth 2.0 web service definition in FME.
Part 2: Create an FME Web Service Definition
Using the credentials from the application created in Part 1, create a new FME Web Service Definition.
1. Create a New Web Service Definition in FME Form
Open FME Form and go to Tools > FME Options > Web Connections. Click Manage Services at the bottom right of the window.
The Manage Web Services window will display a list of the current web services available in FME. You can see the definition of each web service by selecting it from the list. The definition will be displayed on the right side of the window.
This list contains out-of-the-box, as well as user-defined web services, with no obvious way to distinguish between them other than the Web Service Name and Connection Description parameters. It is important, therefore, to thoughtfully name and describe your user-defined web services upon creation so that you can easily distinguish them.
Most of the web service definitions provided with FME are meant to serve as pre-formatted templates that can be copied and modified to create a user-defined web service more quickly. Click on a web service and read the Web Service Setup Instructions.
If there is a web service template available in the list, then click the “+” symbol below the list of web services, choose Create From, and then choose the web service template you want to copy.
If there is no web service template available, then you will need to create a new, empty OAuth 2.0 web service. Click the “+” symbol below the list of web services and choose OAuth 2.0 Service (Authorization Code Flow).
Authorization Code is the most common OAuth 2.0 grant type and the only one currently supported by FME OAuth 2.0 Web Services. There is also a Client Credentials grant type that is still considered OAuth 2.0, but follows a very different authentication flow. If your web service provider uses the OAuth 2.0 Client Credentials grant type, choose Token Service from the list instead. Please consult your web service provider’s documentation. For an example of setting up a token service, please see Creating a Token-Based Web Service in FME.
2. Populate the Web Service Definition
This step will go through populating the web service definition section by section. These instructions can also be used to help adjust the parameter values in the web service definition templates, as needed.
The Overview Section
Enter a meaningful Web Service Name for your web service to distinguish it from the other web services on the list. It’s recommended to include the application name you registered on the web service’s authorization server in Part 1 in the Connection Description, so that it can be cross-referenced. You can also add external help links, for example, to documentation maintained by your organization.
Client Information Section
Enter the client ID and client secret that was obtained in Part 1. If your application was not issued a client secret, check the box next to Optional beside the client secret text box. Enter the same Redirect URI used earlier during application registration in Part 1.
The Authorization Parameters Section
The first step in obtaining access to a web service via OAuth 2.0 is to get authorization (consent) from the user. In FME, this is accomplished by displaying the web service provider’s authorization interface to the user. The Authorization Parameters section contains the URL which points to this interface. This URL is typically found in the developer documentation for the web service provider, most often in a section titled “Authentication” or “OAuth 2.0”. If these titles aren’t present, check in the “Getting Started” section, or use a document search feature (ctrl + F, for example) to perform a keyword search for relevant information.
If you started with a template web service, you often don’t need to modify the Authorization URL. Occasionally, the URL will require specific parameters appended at its end to limit the scope (permissions) or to specify the response type of the authorization request. The Google suite of web services, for example, has these additional authorization URL requirements. Check the web service provider’s documentation for these additional parameters and how to format them correctly.
Retrieve and Refresh Token Parameters Sections
The Retrieve Token Parameters section is where you configure access token acquisition from the web service API. If you started with a template web service, you often don’t need to modify this section. Otherwise, you will need to know the web service API endpoint you wish to access for token retrieval. This endpoint URL can typically be found within the web service provider's API documentation under “Obtaining an Access Token” or similar. Once this parameter is properly configured, an access token will be generated upon user authorization. This token is then used by FME to make secure API calls on behalf of the user, limited to the scopes specified by the user, until the token expires or is revoked. Essentially, the access token lets the web service API know that FME has the user’s permission to access certain protected resources and perform certain authorized tasks.
The Refresh Token Parameters section controls how a new access token is generated from the web service API once the original access token has expired or is revoked. If you started with a template web service, you often don’t need to modify this section. It is important, particularly if publishing the workflow to FME Flow, to configure this parameter correctly so that a new access token can be generated when needed. As long as the refresh token remains valid, new access tokens can be acquired. The Request Format parameter, just below the URL, allows you to specify the desired request format. Typically, the default values are appropriate.
Sometimes the Refresh Token URL is the same as the Retrieve Token URL, while other times, the web service has a dedicated refresh workflow. If there is a dedicated refresh workflow, then a refresh token will have been issued when the original access token was issued. It is best practice to consult the web service provider's API documentation on refresh tokens for further instructions.
3. Test Your OAuth 2.0 Web Service Definition
Now, you are ready to test your newly configured web service definition. At the bottom of the New Web Service template, click Apply to save the web service definition. Then click Sign In With <web service name> or Test. The text on this button may vary with the web service you are configuring.
FME Form will attempt to connect to the web service and show its authorization web page. Typically, this is a prompt for user credentials. Make sure to enter user credentials that have access to the web service resources you plan to access in FME.
FME Form will display the progress of the authorization attempt, including any errors, in the translation log window. If this window is hidden, enable it in FME Workbench under View > Windows > Translation Log. Once you have authorized your web service without errors, click Close at the bottom of the Manage Web Services window and OK in the FME Options window. If you encounter errors, please see the article FME Form Troubleshooting: Web Services and Connections.
The user credentials you enter are only used to authorize the web service and obtain the authorization and refresh tokens. It is these tokens that are stored in FME and used to authenticate after the web service has been authorized - not user credentials.
Part 3: Create a FME Web Sevice Connection
A web service cannot be used on its own to access target resources; it needs to be referenced from a web connection. Many web connections can be created from a single web service. A web connection can be created in Tools > FME Options > Web Connections, but it can also be created from the reader, writer, or transformer where you’re planning to use it. This latter method is described below with the HTTPCaller.
1. Create a new FME Workspace.
Add a Creator transformer and then attach an HTTPCaller transformer to the Creator. Now right-click on the HTTPCaller and select Connect Loggers. You should see two new transformers attached to the HTTPCaller. These Logger transformers will log the results of the API calls made by the HTTPCaller.
2. Add the Web Service API Endpoint to the HTTPCaller
Make sure that you know the API endpoint of the web service you wish to test, and understand what it returns. Double-click the HTTPCaller to open its parameters. Set the HTTP method to the method required for the connection test (GET, POST, etc). Set the Request URL to the desired API endpoint to access the target web service.
3. Add a Web Connection
Enable Use Authentication and select Web Connection for the Authentication Method. In the Web Connection parameter, select Add Web Connection from the drop-down menu.
In the Edit Web Connection dialog that opens, choose the OAuth 2.0 web service you created and enter a Connection Name. Click Authenticate Or Sign in With <web service name> at bottom-right to create the web connection. The text on this button will vary with the web service being used. Authorize the web service with your credentials. This will authorize FME to act on your behalf in accessing and retrieving resources from the web service.
Part 4: Test the Web Service Connection
Confirm that the new web service connection works.
1. Run Workspace
Run your workspace to test the new web service connection as part of a translation. Check the translation log to make sure there are no errors.
2. Inspect the Output
Inspect the output of the HTTPCaller in the Visual Preview window to verify the results are as expected. An _http_status_code value of 200 indicates that the request was successful.
3. Save Workspace
Once you have verified your web service connection, save your workspace. This workspace can serve as a Test Workspace if you choose to share it on the FME Hub.
Part 5: Using a Web Service Connection on FME Flow
Before a web service connection can be used on FME Flow, it must be created in FME Form. Ensure that your connection has been created and tested as per Parts 1 to 4 in this article before continuing.
1. Update the Web Service Application for FME Flow
In Part 1, you registered a new application with the web service provider and entered a redirect URI for FME Form. The application will need to be updated for FME Flow.
Open the application you previously registered. If your web service requires you to specify an application type, you may need to register a new application for FME Flow with a type of Web Application, or equivalent. You may also be able to add a new type to the existing application. Consult your web service provider’s documentation.
Applications often allow for multiple Redirect URIs to support connections from different client applications. The redirect URI used for FME Form will not work for FME Flow. For FME Flow, the Redirect URI will be <your FME Flow URL>/fmeoauth. For example, https://www.cityofsafeville.gov/fmeoauth.
Add a Redirect URI for FME Flow and save the application.
2. Upload to FME Flow
There are many ways to upload a web service connection to FME Flow from FME Form:
- Right-click the web connection name in Tools > FME Options > Web Connections, then choose Upload. This will upload both the web connection and web service.
- Click Upload on the web service definition in Tools > FME Options > Web Connections > Manage Services and then create a web connection on FME Flow.
- Select the web connection in the publish wizard when publishing a workspace in FME Workbench. This will upload both the web connection and web service.
This article will look at the last method, since you have already created a test workspace in the prior FME Form instructions.
In FME Workbench, choose the FME Flow instance that you want to publish to, ensure the FME Flow connection is valid (green check mark) and then click publish. If you have never connected to FME Flow from FME Workbench, please review the documentation found here.
This will launch the publish wizard. After choosing a repository for your test workspace, you will be given the option to select any connections you want to upload. Make sure the web connection is checked, and then click Next.
Leave the default Job Submitter service and click Publish. Along with uploading the workspace file to FME Flow, the web connection and its web service definition will also be uploaded.
If a web service with the same name already exists on FME Flow, only the web connection will be uploaded. If a web connection with the same name already exists on FME Flow, you will be warned.
Check the Translation Log to confirm that the workspace was published successfully.
3. Update the Web Service Definition on FME Flow
Log onto FME Flow as the same user you connected and published with in step 2, or a superuser.
In the FME Flow menu, expand Connections & Parameters and then click Web Connections.
You should see the web connection you published, and its web service, in the list. Click Manage Web Services.
Click the web service name.
Enter the Client Id and Client Secret from the application you updated in step 1. If you needed to register a new application, then you will have a new Client Id and Secret different from the ones used when in the web service definition on FME Form. Enter the Redirect URI - it must match the Redirect URI entered in the application in step 1. The Redirect URI will be <your FME Flow URL>/fmeoauth. Click OK to save the changes.
4. Authorize the Web Connection on FME Flow
Return to the list of Web Connections and click the web connection name.
Click Sign In With <web service name> or Authorize. The text on this button may vary with the web service you are configuring. FME Flow will attempt to connect to the web service and show its authorization web page. Typically, this is a prompt for user credentials. Make sure to enter user credentials that have access to the web service resources you plan to access in FME Flow.
After authorizing successfully, the web connection will have a green checkmark beside it. Click OK.
The authorization can fail for many reasons, including network proxies and mismatched redirect URIs. If you encounter errors, please see FME Flow Troubleshooting: Web Connections.
5. Share the Web Connection
Other FME Flow users will not be able to see or use the web connection until it has been shared with them. If you want others to access the web connection, click the share button and choose the roles or users to share with.
Share Your Web Connection on FME Hub (Optional)
Many members of the FME Community choose to share their web service definitions in the FME Hub. The FME Hub is a great place to find new custom transformers, workspace templates, web connections, and other creative projects that FME users have shared. More information on the FME Hub can be found in this article.
If you would like to contribute your OAuth 2.0 web service definition, first log in with your Safe account and search the FME Hub to make sure a web service definition does not already exist for the service you would like to share. You can apply a Web Connections filter to your keyword search to make the results easier to parse.
Once you have verified that your web service definition does not already exist, return to FME Workbench. Go to the list of web services found under Tools > FME Options > Web Connections > Manage Services.
Find your newly created web service definition in the web service list, along the left side of the Manage Web Services dialog. Select it then scroll to the bottom of the web service definition and click Export.
Choose a location to export to and uncheck Include Credentials, so that other FME users can provide their own. Click OK to export the web service definition as XML.
With your XML export and your Test Workspace (Created in Part 3), return to FME Hub in your browser. Follow the documentation here to upload the web service definition to the FME Hub.
Using the FME Hub OAuth 2.0 Web Service Definitions
Another option for populating a web service definition for a service of interest is to download one of the many formatted OAuth 2.0 web service definitions available from FME Hub. Performing an FME Hub search for "OAuth 2.0 web services", filtered by Web Connections, will return several available OAuth 2.0 web service definitions. Select the definition of interest, and then select Download on the page that opens. You will receive an XML document containing the necessary specifications of the web service definition.
Return to FME Workbench and access the Manage Web Services dialog from the Web Connections menu of FME Options (Tools > FME Options > Web Connections > Manage Services button at the bottom-right). Below the list of web services therein, click on the "+" symbol and choose Import From File. In the dialog that opens, click on the ellipses next to the File to Import text box. Navigate to your downloaded XML web service definition, and click Open. Unless the FME Hub description of the web service explicitly states a password, you can ignore the Password parameter and click OK.
Your newly uploaded web service definition should open. Follow the instructions given in the FME Hub description of the web service to adjust the template as needed. You will likely have to register your own app with the web service provider and enter your own Client Information parameters. You can save and test this web service definition using the same methods discussed in Parts 3 and 4, above.
OAuth 2.0 Web Service Connection Articles
Not all articles listed below will include steps for FME Flow. They can be used in conjunction with this article.
- Airtable
- Autodesk Docs and BIM360
- Cityworks
- Esri ArcGIS Online
- Esri ArcGIS Portal
- Infor
- Microsoft SharePoint
- Microsoft Graph (Microsoft Teams)
- Salesforce
Comments
0 comments
Please sign in to leave a comment.