Introduction
Hypertext Transfer Protocol (HTTP) is an application protocol for exchanging and communicating hypermedia(link) resources between a client and server. It is the main system for exchanging information across the web.
The HTTPCaller transformer empowers you to construct all of your requests, both simple or advanced, and connect to web resources from within your FME workflows.
This article will cover a basic overview of HTTP requests, responses, and methods and how they relate to the HTTPCaller transformer.
Learning About HTTP Requests and Responses
Part 1: Requests
Let’s review how a typical HTTP request is constructed. The core components and composition of a request are.
1. A start line with three parts:
b. A Universal Resource Identifier (URI) defines where the target resource is located. You might be most familiar with a Universal Resource Locator (URL), a type of URI that begins with the network protocol that defines how to access the resource, such as “https” or “ftp”.
c. The HTTP version.
2. Headers to pass name/value pairs with additional information about the request.
3. An empty line to indicate the end of the header fields.
4. An optional message body to include additional information.
Each respective part of a request is customizable from the HTTPCaller’s graphic interface. The HTTPCaller will construct the request for you; you just have to fill in the parameters.
Locator and Method
The Request URL and Request Method indicate the resource location and request type to send to the server, respectively. Query string parameters that would otherwise be a part of the URL can be input using a table. The transformer applies any URL encoding to entries.
HTTPCaller will always use the latest version of HTTP, so there is no need to specify.
Headers
The Headers table can be used to create custom HTTP headers for a request. The Name column supplies a dropdown list of commonly used headers, or create your own by typing them in or using attributes.
Message Body
A message body can be formatted to send content with an HTTP request. Select an upload body, use a file, or multipart/form data. The Content Type header can be set alongside.
Some fields are made available or hidden based on the HTTP method selected. For example, GET requests don’t use a message body. After selecting GET, the Body parameter becomes unavailable.
Part 2: Responses
Sending a request to a URL elicits a response from the server. Response messages are constructed in much the same way as a request. They generally contain a response body, headers, a status message, and status code. HTTPCaller has options for handling and storing responses to suit your workflows.
1. A status line with three parts:
b. The status code to inform on the success or failure of the request.
c. The status message with a description of the status code.
2. Headers to pass name/value pairs with additional information about the response.
3. An empty line to indicate the end of the header fields.
4. An optional message body to include additional information.
Message Body
In Workbench, the returned response body can be saved to an attribute or file. Choosing File prompts more menu options for storing the message on your desktop. This is useful if you are downloading a file from the web service. This file can be read into the workspace after the HTTPCaller with a FeatureReader.
Saving the response body to an attribute stores the message to an attribute _response_body by default. It is viewable in Visual Preview and easily passed downstream to be used in the workflow. Use Response Body Encoding to specify the character encoding of the message. If unfilled, it’ll be detected from the headers returned with the HTTP response.
Status and Headers
When a List Attribute is specified, returned headers will be stored with the feature, viewable in the Feature Information window or usable in the workflow.
Status codes returned with an HTTP response are saved to the _http_status_code attribute by default. For unsuccessful requests, the _error attribute contains a brief error message. This allows you to easily confirm successful requests or troubleshoot unexpected responses.
Part 3: A Deeper Dive Into Request Methods
The HTTPCaller supports many of the common operations for accessing URLs: GET, PUT, POST, DELETE, HEAD, PATCH, OPTIONS, COPY, LOCK, MKCOL, MOVE, PROPFIND, PROPPATCH, UNLOCK, CHECKOUT, CHECKIN, UNCHECKOUT, LINK, UNLINK.
Sometimes HTTP methods are referred to as verbs. While most methods are designed to take some action on the specified resource ( e.g. GET, PUT, POST, DELETE, PATCH), this is misleading because some are better classified as nouns (e.g. HEAD, OPTIONS).
There is some key terminology to cover that will help you understand and use the different methods effectively.
When a method is idempotent, it means that an identical request can be made several times with no effect after the first request. For example, you might use DELETE to remove a resource with your first call, but repeat calls will not remove the same resource again.
Methods can also be either safe or unsafe. A safe method is read-only, it does not alter the state of the server or resources.
| Request | Action | Idempotent | Safe |
|---|---|---|---|
| GET | Retrieves representation of the resource requested | Yes | Yes |
| PUT | Sends data or requests a change to the server. Repeating the same request successively has no effect. | Yes | No |
| POST | Sends data or requests a change to the server. Repeating the same request successively may have repeated effects. | No | No |
| DELETE | Deletes a specified resource. | Yes | No |
| HEAD | Requests the headers that would be returned with an identical GET request. | No | No |
| PATCH | Partially updates or modifies a resource. | No | No |
| OPTIONS | Requests the permitted options at that resource. | Yes | Yes |
It is notable that not every server is configured to support every method. Furthermore, methods may be used for different purposes, depending on the configuration and design of the server. It is always best to consult the documentation.
Optimizing HTTP Requests Using the HTTPCaller
Concurrency
Making multiple HTTP requests can slow down a workspace that would otherwise be quite fast. The Maximum Number of Concurrent HTTP Requests allows a user to make multiple requests at the same time. Let’s say you are sending 100 requests to the HTTPCaller, if the concurrent requests are set to 25, FME will submit 25 requests at a time for a total of 4 times. If the concurrent requests are set to 1, FME will submit 1 request at a time for a total of 100 times. The more requests you submit at one time, the more useful this parameter will be!
Please note that not all web services will support the concurrent requests and reject them. If there is an issue, set the concurrent requests to 1 or try rate-limiting as mentioned below.
If you are using an HTTPCaller in a custom transformer with a loop, set the concurrent requests to 1. Looping will not work with concurrent requests.
Rate Limiting
APIs often have rate limits. These will prevent a user from submitting too many transactions at one time, overwhelming the web service. In most cases, you will not need to use the rate-limiting option in the HTTPCaller. However, if you start to see errors such as 429- Too Many Requests error from the API, then you can use the rate-limiting feature. Typically, your API documentation should outline rate limits that can be applied in the HTTPCaller.
Retry Failed Requests
Requests can automatically be retried by the HTTPCaller. This can be helpful if you encounter network issues from the web service you are connecting to. Network issues can arise unexpectedly; therefore, by retrying the requests, we can help mitigate these issues before they arise.
Additional Resources
- How to Access an API using the HTTPCaller
- Making a GET Request with HTTPCaller
- Leveraging the Box.com Web Service with FME
- Getting Started with FME and the Web
- HTTPCaller and Google Calendar API: GET
Comments
0 comments
Please sign in to leave a comment.