Optimizing Large Datasets for KML and Google Earth | Creating a KML Network Link File

Introduction

Splitting large datasets into smaller pieces is useful for the display of KML in Google Earth. A single workspace creates both the KML Network link file and the tiled KML datasets. The first part of the workspace uses grid features and creates tile boundaries. The second part of the workspace reads the source data, styles it for KML, and writes it out to KML tiles.
 
This sample illustrates how to optimize large datasets for display in Google Earth using KML. The workspace illustrates:

• Using Regions (tiling)
• KML Network Link
• Use of the SchemaMapper to set the KML style attribution - an alternative to using several KMLStyler transformers, and Dynamic workflows to read & write the data.

If you attempt to display a large dataset into Google Earth it will likely freeze. You can work around this by pre-processing the dataset into smaller pieces (tiles) and then serving them. This also means it's easier to keep the data up-to-date since you only have to update an individual tile - not the entire dataset. The approach used here involves requesting data for a smaller area as you zoom in using Google Earth.
 

Step-by-step Instructions

Part 1: Create Overview Tile

1. Add Data
Open FME Workbench, and start a blank workspace. Add an Autodesk MapGuide Enterprise SDF reader to the canvas and browse to the InterpolisDatabase.sdf dataset, which is available to download from the Files section on this article. Set the Workflow Options to Single Merged Feature Type, then open the Parameters. 

 
In the parameters, enable Remove Table Qualifier, then click on the ellipsis next to Tables. Enable Select All, then click OK three times to finish adding the reader. 

 
2. Filter Features into Grid
Next, we are going to filter the data based on whether its feature type name is CityGrid. We will be using the CityGrid features to define our area of interest, as well as to create the tiles and an overview tile. Add a FeatureTypeFilter to the canvas and connect it to the <All> reader feature type. In the parameters, add CityGrid to the list of Feature Type Names, then move it to the top so it will become the topmost output port. 

 
3. Calculate Area of Each CityGrid
We want to remove any tiles that are too small, so first let’s calculate the area of each CityGrid. Add an AreaCalculator to the canvas and connect it to the CityGrid output port on the FeatureTypeFilter. We will leave the default parameters. 


4. Remove Small Tiles
Using the Tester, we will now remove any tiles smaller than 10,000,000. Add a Tester to the canvas and connect it to the AreaCalculator. In the parameters, set up the following test:

_area > 10000000

(That is 7 zeros)


5. Reproject Data
Google Earth and the OGC KML format only accept the latitude and longitude coordinate system. Add a CsmapReprojector to the canvas and connect it to the Line output port on the LineCombiner. In the parameters, set the Destination Coordinate System to LL84, then click OK. 

 
6. Create Bounding Box
Now we will generate a bounding box from the CityGrid features. Add a BoundingBoxAccumulator to the canvas and connect it to the CsmapReprojector. We will accept the default parameters. 
 
7. Create Published Parameter for Output Directory
Because we will be referencing the output directory for each of our Network Links, let’s create a published parameter to make this easier. In the Navigator pane, right-click on User Parameters then select Manage User Parameters. 

In the Manage User Parameters dialog, click on the green plus sign (+), then select File/URL. Then set the following properties for the parameter:

• Parameter Identifier: KMLDIR
• Prompt: KML Output Directory
• Published: Enabled
• Required: Enabled
• Disable Attribute Assignment: Disabled
• Expand File/URL Configuration:
  • Items to Select: Folders
  • Access Mode: Write
• Default Value: $(FME_MF_DIR)\OutputKML

 
8. Create Attributes for Overview Tile
There are several attributes we will need to create to tell the output file that this is a Network Link KML. Add an AttributeCreator to the canvas and connect it to the BoundingBoxAccumulator. In the parameters create the following attributes. 

9. Set Region for Overview Tile
Next, we need to set the display extents for the overview tile, for this, we will use the KMLRegionSetter. Add a KMLRegionSetter to the canvas and connect it to the AttributeCreator. We can accept the default parameters. The default is set to a minimum display size of 256 pixels, which means that one must be zoomed in to at least 256 pixels to view the overview tile in Google Earth. 
 

Part 2: Create Links to Tiles

10. Create Links to Tiles
With our overview tile set up, we need to create the links that we will use once we create the tiles containing data. Add another AttributeCreator to the canvas and connect it to the CsmapReprojector. 


In the parameters, create the following attributes:

 
11. Set Region for Tiles
Next, we need to set the display extents for each of the tiles. Add a KMLRegionSetter to the canvas and connect it to the AttributeCreator. In the parameters, set the Minimum Display Size to 512 and then click OK. This means the user must be zoomed in to at least 512 pixels to view the individual tiles in Google Earth. 

 
12. Write Out Network Link and Tiles
The network link and tiles are ready to be written out. Add an OGC/Google KML writer to the canvas, browse to a location to save the dataset. Name the KML Output.kml, we will be changing this output name in a moment. Set the Feature Type Definition to Manual, then click OK.


In the Feature Type dialog, set the Feature Type Name to NetworkLink, then click OK. This feature type must be named NetworkLink so that Google Earth recognizes it as a network link dataset. 


Connect the NetworkLink writer feature type to both the KMLRegionSetters. 


Now we need to set up the dataset fanout to use the _networkLinkFile attribute we created. In the Navigator pane, expand the Output writer then right-click on the Destination Google KML File, and select Link to User Parameter. 


In the Set User Parameter dialog, select the KMLDIR user parameter then click OK. Next, double-click on Fanout Dataset in the Navigator pane. 


In the Edit Output [OGCKML] dialog, enable Fanout Dataset, then set the Fanout Expression to:

@Value(_networkLinkFile).kml

Part 3: Create KML Tiles with Features

In this part, we are going to be creating the KML tiles using the TILE_NAME attribute from the CityGrid, and then adding features to it. 
 
13. Keep TILE_NAME Attribute
For the remainder of the workspace, we only need to refer to the tiles by using the TILE_NAME attribute from the CityGrid feature type. Add an AttributeKeeper to the canvas and connect it to the Passed output port on the Tester. 

 
 In the parameters, select TILE_NAME as the Attribute to Keep. 

 
14. Extract Boundary from Tile
Now we need to extract the coordinates for the bounding box of each tile. Add a BoundsExtractor to the canvas and connect it to the AttributeKeeper. In the parameters, remove the attributes for minimum and maximum z. 

 
15. Reproject Attributes
When you extract the bounds of a feature, it presents the values in the same coordinate system as the dataset, so we will need to reproject it to LL84. Since we have both max and min values, we will need to perform this step twice. Add an AttributeReprojector to the canvas and connect it to the BoundsExtractor. In the parameters, set the X Attribute to _xmax and the Y Attribute to _ymax. Next, set the Source Coordinate System to TX83-CF and the Destination Coordinate System to LL84. 

 
Add a second AttributeReprojector to the canvas and connect it to the first AttributeReprojector. In the Parameters, set the X and Y Attribute to _xmin and _ymin, respectively. Then set the Source and Destination the same as the previous transformer. 
 
16. Create Area of Interest
Now we can create our Area of Interest, to do this we will use the Clipper to clip all the other feature types with the CityGrid. Add a Clipper to the canvas, connect the Clipper input port to the AttributeReprojector_2 Output port, then the Clippee input port to the <Unfiltered> output port on the FeatureTypeFilter. 

In the parameters, enable Merge Attributes, set the Accumulation Mode to Prefix Clipper, then set the Prefix to clipper_ 

 
17. Create clipper_TILE_NAME Attribute
To match the output attributes coming from the Clipper, we need to create an attribute coming from the tile. Add an AttributeCreator to the canvas and connect it to the BoundsExtractor. 


In the parameters, create a New Attribute called clipper_TILE_NAME then give it the value of Overview. 

 
18. Set KML Properties
When we view the dataset in Google Earth, we will want to be able to filter by each dataset (feature type). Add a KMLPropertySetter to the canvas, and connect both the AttributeCreator_3 (coming from the BoundsExtractor) and the Inside output port on the Clipper. 

 
In the parameters, set the Navigation Tree Name to fme_feature_type and click OK. 


19. Filter Feature Types to Set Region For
To make this example easier, we will only set the region for a couple of feature types for demonstration purposes. Add another FeatureTypeFilter to the canvas and connect it to the KMLPropertySetter. In the parameters, filter out Buildings and WaterMeters. 

 
20. Set Region for Buildings and WaterMeters
Add another KMLRegionSetter to the canvas and connect it to both the Buildings and WaterMeters output port on the FeatureTypeFilter_2. In the parameters, set Calculate from Bounding Box to No, then set the Minimum Display to 2048. Expand the Bounding Box section and set the following parameters:

• Minimum X (West): _xmin
• Minimum Y (South): _ymin
• Maximum X (East): _xmax
• Maximum Y (North): _ymax

21. Read Schema for Styling
To finish styling the Buildings and WaterMeters, as well as the other features we will use a predefined schema. Add a SchemaMapper to the canvas and connect it to the KMLRegionSetter_3 (coming from the FeatureTypeFilter_2) and the <Unfiltered> output port on the FeatureTypeFilter_2 (filtering Buildings and WaterMeters). 


In the SchemaMapper parameters, set the Format to Comma Separated Value (CSV), then browse to the SchemaMapping.csv file. Click Next. 


Now we will create a series of Actions to define the schema rules. For more information see the SchemaMapper Tutorial. 
First, create a Filter action by clicking on the plus sign at the bottom. In the Create Filter dialog, set the Attribute Name Field to SourceFeatureTypeAttribute and set the Attribute Value Field to SourceFeature, then click OK. 


Next, create a second Filter action with the Attribute Name Field set to FilterAttribute and the Attribute Value Field set to FilterAttributeValue. 
Then, create a Map Feature Types action, set the Source Feature Type Field to SourceAttribute, and the Destination Feature Type Field to KMLAttribute. 
Finally, create a New Attribute action, set the Attribute Name Field to FormatAttribute and the Attribute Value to FormatAttributeValue. 
Click Finish to save the SchemaMapper parameters.

 
22. Write Out Tiles
Add another OGC/Google KML writer to the canvas and browse to the same folder as the previous writer. Name this dataset Output.kml and then change the Feature Type Definition to Dynamic (Advanced) then click OK. Note that the Dynamic feature type definition automatically changes the Feature Type Name to fme_feature_type and we will be setting the feature type fanout in a moment, so the output name of Output.kml doesn't matter. 


Connect the <Dynamic> writer feature type to the SchemaMapper Mapped output port. Then in the Navigator pane, expand the second Output [OGCKML] - 2 writer then right-click on the Destination Google KML File and select Link to User Parameter. In the Set to User Parameter dialog, select the KMLDIR parameter. 


Next, still in the Navigator pane, double-click on Fanout Expression. Set the Fanout Expression to: 

TILE_@Value(clipper_TILE_NAME).kmz

 
23. Run Workspace and View Output 
Run the workspace and then open the output folder. There will be three different types of files:

• KML_Network_Link.kml - Links to all of the tiles
• TILE_Overview.kmz - Overview tile for the area of interest
• TILE_<tile name>.kmz - KML regions (tiles) with features

Open the KML_Network_Link.kml dataset in Google Earth. When opening the network link, it will display the grid TILE_Overview.kmz dataset as we had set the region to 256 pixels. 

 
Now if you zoom into a tile or double-click on a tile in the Places pane, it will zoom in and show more detail. The tile below is showing the detail for L27. 

 
If your computer is having trouble viewing this file in Google Earth, uncheck all but the tile you are trying to view in the Places pane. 

Data Attribution

The data used here originates from data made available by the Government of Austin, Texas. It contains information available to the public domain.