Introduction
The GeoNames database is a free, crowdsourced geospatial resource containing over 12 million unique features worldwide, organized into 9 feature classes. These features are readily accessible via several web service endpoints to those with a (free) valid user account. This article provides a demo of the Weather web service offered by GeoNames, and shows a use case for the JSONExploder and JSONExtractor transformers.
Requirements
As implied above, you must sign up for a free GeoNames user account before using these web services. Once you submit your request for a new user account, you must confirm it using the link sent to the email address you used during account registration.
Next, log in to GeoNames using your newly verified account credentials, click on your user name at the top right, and enable the free web services at the bottom of the user profile page. Once enabled, the web service APIs become available for your use.
Step-by-Step Instructions
The GeoNames web service of interest for this demo is the Weather Service. We will tailor the example shown under the Weather Stations to the most recent Weather Observation section of the Weather Service documentation to retrieve the most recent observations from weather stations located within the continental United States. The retrieved weather data will be used to build a series of contours based on temperature rather than elevation to visualize the most recent temperature readings across the continental United States.
Please download the attached template, WeatherMap_2022_1.fmwt, open it in FME Workbench, and follow the instructions in the article. The workspace contains numbered bookmarks matching each step described below.
1. Read In the Bounding Box Data
The first step in this process is to provide a bounding box feature, which will restrict the results of the weather service API call to a focal area. Essentially, this bounding box dataset serves the same purpose as a spatial filter. In this demo, the bounding box is an area feature representing the continental United States. This feature can be replaced with any 2D bounding box feature as required, including a polygon derived from a Creator transformer.
2. Create a Linear Feature from the Bounding Box Polygon
The bounding box is used in two steps of the demo workspace. In this step, the bounding box geometry is converted from a polygon to a linear feature using the GeometryCoercer transformer. The end result of this operation is a linear outline of the original bounding box, which will be used in a later step to extract contours.
3. Build the URL to Pass to the GeoNames Weather Service
In this step, the irregular bounding box feature is converted to a square feature by the BoundingBoxReplacer transformer. This new bounding-box square is passed through the Buffer transformer, which applies a buffer around the input to ensure the entire area of interest is captured. Next, the BoundsExtractor transformer extracts the minimum and maximum x-, y-, and z-coordinate values and assigns them as new attributes of the buffered bounding box.
These new attributes are passed to the Concatenator transformer as the north, east, south, and west parameters used to build the URL for the HTTPCaller transformer. The full URL can be viewed by selecting the gear icon of the Concatenator and viewing the Concatenated Result in the parameters dialog that opens.
A few key things to note about this URL:
- The GeoNames Weather API endpoint is http://api.geonames.org/weatherJSON
- The extracted x- and y- min/max values are included in the URL as north, south, east, and west parameters
- A limit of 500 records has been set. This limit can be adjusted as needed
- The required username parameter is derived from a published parameter prompted at runtime. Make sure your workspace is set to Prompt for User Parameters (under the Run tab), and enter the GeoNames username you set for your GeoNames user account.
The concatenated URL is stored as the new _url attribute of the buffered bounding box feature.
4. Send the URL as an HTTP GET Request to the GeoNames Weather Service
Use the HTTPCaller transformer to send the GET request using the _url attribute created in the previous step.
As you may have noticed, the API endpoint for this web service finishes with the text JSON. This means that the results of an API call to this web service are sent back in JSON format. This demo saves this JSON response as an attribute called _url_contents.
5. Extract Attributes from the JSON Response Body
The attribute containing the JSON response (_url_contents) is, by default, a list, since FME treats JSON-formatted attributes as lists. You can see the contents of the JSON response by partially running the workspace to the HTTPCaller, viewing the cached results in Visual Preview, and then selecting the ellipses next to the _url_contents value.
Currently, the _url_contents attribute value is challenging to interpret. We will use the JSONFragmenter transformer to extract information from this JSON list attribute, making this information more accessible for further processing. This transformer will split the attribute into separate features based on the value of the JSON Query parameter.
Click the gear icon for this transformer to view its parameters. In particular, note the value of the JSON Query parameter:
json["weatherObservations"][*]
This query specifies that all elements (denoted by the * wildcard) one level down from the “weatherObservations” element should be split out as separate features. In the example JSON text below, individual elements within the “weatherObservations” element are denoted by { }. The first element that is to become a separate FME feature is shown in bold; the next feature is underlined.
"weatherObservations":[{"clouds":"n/a","weatherCondition":"n/a","observation":"K1P1 190540Z AUTO 00000KT 10SM CLR 01/00 A3004 RMKAO2","windDirection":0,"ICAO":"K1P1","lng":-71.75,"temperature":"1","dewPoint":"0","windSpeed":"00","humidity":93,"stationName":"Plymouth,Plymouth Municipal Airport","datetime":"2008-04-19 07:40:00","lat":43.7833333333333},{"clouds":"n/a","weatherCondition":"n/a","observation":"KAFP 190540Z AUTO 21004KT 10SM CLR 16/07 A3002 RMKAO2","windDirection":210,"ICAO":"KAFP","lng":-80.0677777777778,"temperature":"16","dewPoint":"7","windSpeed":"04","humidity":55,"stationName":"Wadesboro, Anson County Airport","datetime":"2008-04-19 07:40:00","lat":35.0169444444444}...
In the demo workspace, running a partial translation to the JSONFragmenter will produce 500 new features, corresponding to 500 individual weather observations. You may recall that the Concatenator transformer’s maxRows parameter was set to 500, meaning we would only expect to see 500 individual weather observations returned from the HTTPCaller. So our results are as expected here.
The consecutive JSONExtractor transformers create new feature attributes from each feature’s corresponding _url_contents list attribute. For example, the JSON Query parameter of the JSONExtractor_3 transformer is set to json["temperature"], and the Target Attribute is set to temperature. This setup will create an attribute named temperature for each weather observation feature and set its value to the temperature recorded at the associated weather station. Click the gear icon for each JSONExtractor to examine each JSON Query. Of particular note are the extracted latitude and longitude values for each weather observation. These coordinate values, corresponding to the location of a weather station, will be used to generate contours in a later step.
6. Format the Date and Filter for Weather Observations No Older Than One Day
In this demo, we are only interested in visualizing the most recent temperature observations from weather stations across the continental United States. Thus, we should filter out any weather observation records older than 1 day. First, the DateFormatter and TimeStamper transformers format the extracted datetime attribute so that other transformers and FME functions can read it. The AttributeCreator and DateFormatter_2 transformers create and populate a _destDate attribute that holds yesterday’s date (relative to the system date). Finally, the Tester transformer filters out any observations older than 1 day by comparing the original observation date (datetime) to _destdate.
7. Create Points for Filtered Weather Observations
The VertexCreator transformer receives the filtered weather observations and generates point data from each observation’s latitude and longitude attributes. Since each observation comes from a weather station, the output shows the locations of the weather stations. The points generated are 3D points, with the temperature attribute set as the z-value.
These points are used in two separate processing streams: one to generate temperature-based labels for weather stations (Step 8) and the other to generate contours (Step 9).
8. Generate Labels for Weather Stations Based on Temperature
The AttributeRounder transformer rounds the temperature values for each station to the nearest degree Celsius. Rounded values are then split by the Tester transformer into two streams: one stream for features with temperature values at or above 0, and a stream for those with temperature values below 0. Positive temperatures are prefixed with a + and then saved to a new _label attribute by the Concatenator transformer, while a new _label attribute is created by the AttributeCopier and applied to the negative temperature stream. Finally, a LabelPointReplacer transformer is used to create label points, using the _label attribute of each input stream as the label value.
9. Create DEM Points From Temperature Z-Values, then Generate and Smooth Contours
The points generated by the VertexCreator transformer are passed to the SurfaceModeller transformer, which generates a series of DEM points based on temperature rather than elevation. These DEM points are passed through a CoordinateFetcher transformer to retrieve their z-values, and then through a Tester transformer to filter out any undefined z-values (a.k.a. qNaN values).
These filtered DEM points then enter the ContourGenerator transformer. This transformer builds lines of equal temperature, called isotherms, from the input DEM points. The output contour lines, however, are jagged and require smoothing; this is accomplished by the Generalizer transformer. This transformer provides many smoothing algorithm options. For this translation, we will use the McMaster(Smooth) algorithm.
10. Extract and Process the Generalized Contours
The bounding box feature is passed to the Intersector transformer, along with the generalized contours, to extract only those contours that fall within its bounds (for this demo, within the continental United States). Next, the AreaBuilder transformer uses the extracted contours to build isotherm area features, or areas of equal temperature. This transformer also creates a list attribute, weatherObs, containing all attributes of the input contour lines, including the temperature z-values.
The ListHistogrammer transformer uses the weatherObs attribute to count how often each temperature value occurs within the contour area dataset and stores the results as a list attribute called _histogram. The ListExploder transformer generates the new attributes count and value for each area feature by exploding the _histogram list attribute. These two new attributes are used to create a color ramp to visualize temperature variations across the continental United States.
11. Build and Apply a Color Ramp for the Isotherm Area Features
A color ramp is helpful for visualizing the gradual changes in a continuous phenomenon across an area of interest. A suitable color ramp will use colors that viewers intuitively associate with the phenomenon of interest. In this step, we will build a red-blue color ramp to show the gradual shifts in temperature across the continental United States. As red is often associated with higher temperatures and blue with lower ones, this color ramp will progress from red to blue as temperature values shift from higher to lower.
First, the StatisticsCalculator transformer computes the minimum, maximum, and range values for the input isotherm areas based on the value attribute created by the ListExploder in Step 10. The two ExpressionEvaluator transformers calculate values for the _red and _blue attributes, which are used to set the RGB fill value for each area. The Concatenator transformer creates a new attribute, fme_color, that contains the _red and _blue values for each isotherm area, separated by a constant value (green).
Lastly, an AttributeCopier transformer copies these RGB-formatted values for each area to the fme_fill_color format attribute. This attribute, when properly formatted as <redvalue>,<greenvalue>,<bluevalue>, controls the fill color that FME applies to area features. In this demo, the RGB values passed to the fme_fill_color attribute all specify colors within the red-blue color range. The result is a series of isotherm areas that gradually change from red through violet to blue as temperatures decrease.
12. Clean Up the Temporary Attributes and Write Out Isotherm Areas
This last step uses the AttributeRemover transformer to remove all the extraneous or unnecessary attributes from the area isotherm features except for fme_color, fme_fill_color, and value. Next, the AttributeRenamer transformer renames the value attribute to temperature, better indicating the phenomenon it represents. Finally, the isotherm area features are written in a format of your choosing. The demo shows a Logger transformer connected to the AttributeRenamer. Replace this Logger with a writer of your choosing.
The resulting isotherm areas should look similar to the following, keeping in mind that the pattern you see in your results will likely differ due to temporal differences in the temperature readings obtained from the HTTPCaller. Notice how there are still areas included in the isotherms that fall outside the bounds of the bounding box. These areas can be removed with a Clipper transformer.
Additional Considerations
Using this template workspace, you can easily replace the bounding box data with an input polygon of your area of interest. Another change to consider is the inclusion of a method for handling temperature outliers. For example, it would be highly unusual to receive an August temperature reading of 30° Celsius from a weather station in the Arctic region of Canada. If this temperature reading is included in DEM point generation, the resulting contours may exhibit greater error. To reduce this error, a Tester transformer could be used to exclude identified outliers before generating DEM points. Finally, if a visualization of temperature shifts over time is required, an ImageRasterizer transformer could be used to generate a series of rasters depicting temperature changes.
Consider this demo workspace as a starting point for exploring and utilizing the variety of web services available through GeoNames. Although this demo is specific to the GeoNames Weather web service, you can use the same general process to form an HTTP GET request for the variety of other web services that GeoNames provides. A look through the GeoNames web service documentation will provide details of the different service endpoints and expected request parameters.