Optimizing Large Datasets for KML and Google Earth | Creating a KML Network Link File

Files

FME2021_TilingLargeKMLs.zip
- 8 MB
- Download
SourceData.zip
- 3 MB
- Download
FME2025.2_TilingLargeKMLs.fmwt
- 3 MB
- Download

Introduction

Splitting large datasets into smaller pieces is useful for the display of KML data in Google Earth. This tutorial achieves this by pre-processing the dataset into smaller pieces called tiles. A KML Network link file and tiled KML datasets are produced. Tile boundaries are created using grid features, and the source data is read, styled for KML, and written out to KML tiles.

This tutorial demonstrates how to optimize large datasets for display in Google Earth with KML, illustrating:

Using Regions (tiling)
KML Network Link
The use of the SchemaMapper to set the KML style attribution - an alternative to using several KMLStyler transformers, and Dynamic workflows to read & write the data.

Another advantage of splitting the dataset into small tiles is that it makes it easier to keep the data up-to-date, since you only have to update an individual tile, not the entire dataset. The approach used here involves requesting data for a smaller area as you zoom in using Google Earth.

The source data and workspace template for this tutorial can be downloaded from the Files section.

Step-by-Step Instructions

Part 1: Create Overview Tile

1. Start FME Workbench and Add City Database GeoPackage

Start FME Workbench, and click on New to open a blank workspace.

To add a source dataset, click the Reader button from the Toolbar. In the Add Reader dialog, enter the following:

Format: OGC GeoPackage
Dataset: /CityDatabase.gpkg
- Click on the ellipsis (blank document icon for macOS) and browse to the dataset

Then click OK.

2. Filter Features into Grid

We want to use the CityGrid features to define our area of interest and create tiles, so we will use a FeatureTypeFilter to filter the data based on feature type.

Type FeatureTypeFilter anywhere on the canvas and press Enter on your keyboard to add it to the workspace. Connect the CityDatabase reader feature type (labeled <All>) to the FeatureTypeFilter Input port.

Double-click the newly added transformer to open the parameters window, and add CityGrid to the list of Feature Type Names.

Feature Type Names: CityGrid

Use the Move Up button to move the CityGrid cell to the top, so it will become the topmost output port.

3. Calculate Area of Each CityGrid with the AreaCalculator

We want to remove any tiles that are too small. First, we will calculate the area of each CityGrid with an AreaCalculator transformer.

Add an AreaCalculator to the canvas and connect the FeatureTypeFilter CityGrid output port to the AreaCalculator Input port.

4. Remove Small Tiles

Using the Tester, we will now remove any tiles smaller than 10,000,000. Add a Tester to the canvas and connect the AreaCalculator Output port to the to the AreaCalculator.

In the parameters, set up the following test:

Left Value: _area
Operator: >
Right Value: 10000000

5. Reproject Data

Google Earth and OGC KML format only accept geographic coordinate systems (latitude and longitude). We will set the coordinate system using a CsmapReprojector transformer.

Connect the Tester Passed output port to a CsmapReprojector Input port. In the parameters, set the coordinate system:

Destination Coordinate System: LL84

6. Create Bounding Box

Now we will generate a bounding box from the CityGrid features. Add a BoundingBoxAccumulator to the canvas and connect it to the CsmapReprojector Reprojected output port. We will accept the default parameters.

7. Create Published Parameter for Output Directory

Since we will be referencing the output directory for each of our Network Links, let’s create a published parameter to make this easier.

In the Navigator pane, right-click on User Parameters, then select Manage User Parameters:

In the User Parameters dialog, click on the green plus sign (+), then select File/Folder/URL:

Then set the following properties for the parameter:

Parameter Identifier: KMLDIR
Prompt: KML Output Directory
Required: Enabled
Show Label: Enabled
Items to Select: Folders
Access Mode: Write
Default Value:
```
$(FME_MF_DIR)/OutputKML
```

Now, when the workspace is run, the user can choose their own output directory, if they do not accept the default.

8. Create Attributes for Overview Tile

There are several attributes we will need to create to tell the output file that this is a Network Link KML.

Add an AttributeCreator to the canvas and connect it to the BoundingBoxAccumulator BoundingBox output port. In the parameters, create the following attributes:

New Attribute	Attribute Value
kml_id	Overview
kml_name	Overview
kml_link_href	$(KMLDIR)/TILE_OVERVIEW.kmz
_networkLinkFile	KML_Network_Link

9. Set Region for Overview Tile

Next, we need to set the display extents for the overview tile, for this, we will use the KMLRegionSetter.

Connect the AttributeCreator Output port to a KMLRegionSetter Input port. We can accept the default parameters.

The default is set to a minimum display size of 256 pixels, which means that one must be zoomed in to at least 256 pixels to view the overview tile in Google Earth.

Part 2: Create Links to Tiles

1. Create Links to Tiles

With our overview tile set up, we need to create the links that will be used once the tiles have been created.

Add another AttributeCreator to the canvas and connect it to the CsmapReprojector.

In the parameters, create the following attributes:

Attribute Name	Attribute Value
kml_id	@Value(TILE_NAME)
kml_name	@Value(TILE_NAME)
kml_link_href	$(KMLDIR)\TILE_@Value(TILE_NAME).kmz
_networkLinkFile	KML_Network_Link

2. Set Region for Tiles

Next, we need to set the display extents for each of the tiles.

Add another KMLRegionSetter to the canvas and connect it to the newly added AttributeCreator (AttributeCreator_2).

In the parameters, set the Minimum Display Size to 512 and then click OK:

Minimum Display Size (pixels): 512

This means the user must be zoomed in to at least 512 pixels to view the individual tiles in Google Earth.

3. Write Out Network Link and Tiles

The network link and tiles are now ready to be written out to OGC/Google KML format.

Click the Writer button in the Toolbar and set the following parameters:

Format: OGC / Google KML
Dataset: \Output.kml
- This will be changed later to the User Parameter we created

Click OK. In the Feature Type dialog, set the feature type name:

Feature Type Name: NetworkLink

This feature type must be named NetworkLink so that Google Earth recognizes it as a network link dataset.

Connect both KMLRegionSetters Output ports to the NetworkLink writer feature type:

4. Link User Parameter

We will now set the output destination to the User Parameter we created earlier (Part 1 Step 7).

In the Navigator pane, expand the Output [OGCKML] writer. Right-click the Destination Google KML File and select Link to a Parameter…

In the Link to a Parameter dialog, link the output to the User Parameter we created, KMLDIR:

Link to: KMLDIR

5. Set Up Dataset Fanout

We want the NetworkLink writer to fanout the dataset using the _networkLinkFile attribute we created.

In the Navigator, under OutputKML, double-click the Fanout Dataset option.

In the Edit Output [OGCKML] dialog, set the parameters as follows:

Fanout Dataset: Enabled
Fanout Expression:
```
@Value(_networkLinkFile).kml
```

Part 3: Create KML Tiles with Features

We will now create the KML tiles using the TILE_NAME attribute from the CityGrid, and then adding features to it.

1. Keep TILE_NAME Attribute

For the remainder of the workspace, we only need to refer to the tiles by using the TILE_NAME attribute from the CityGrid feature type.

Add an AttributeKeeper to the canvas and connect it to the Passed output port on the Tester.

In the parameters, set the Attributes to Keep to TILE_NAME:

Attributes to Keep: TILE_NAME
- Click the ellipsis to choose from the list of attributes

2. Extract Boundary from Tile

To get the coordinates for the bounding box of each tile, we will extract the coordinates with a BoundsExtractor.

Connect the AttributeKeeper Output port to a BoundsExtractor Input port and remove the Z attribute values:

Min Z Attribute:
- Delete existing value to be blank
Max Z Attribute:
- Delete existing value to be blank

3. Reproject Attributes

When you extract the bounds of a feature, it presents the values in the same coordinate system as the dataset, so we will need to reproject it to LL84.

Since we have both max and min values, we will need to perform this step twice. Add an AttributeReprojector to the canvas and connect it to the BoundsExtractor Output port. In the parameters, set the following:

X Attribute: _xmax
Y Attribute: _ymax
Source Coordinate System: TX83-CF
Destination Coordinate System: LL84

Add a second AttributeReprojector to the canvas and connect it to the first AttributeReprojector. In the Parameters, set the following:

X Attribute: _xmin
Y Attribute: _ymin
Source Coordinate System: TX83-CF
Destination Coordinate System: LL84

4. Create Area of Interest

Now we can create our Area of Interest. To do so, we will use the Clipper to clip all the other feature types to the CityGrid.

Add a Clipper transformer to the workspace. Connect the AttributeReprojector_2 Output port to the Clipper Clipper port. Connect the FeatureTypeFilter <Unfiltered> output port to the Clipper Candidate input port:

Set the following parameters in the Clipper:

Merge Attributes: Enabled
Accumulation Mode: Prefix Clipper
Prefix: clipper_

5. Create Clipper_TILE_NAME Attribute

To match the output attributes coming from the Clipper, we need to create an attribute coming from the tile.

Add an AttributeCreator to the canvas and connect it to the BoundsExtractor. Create the following attribute and assign its value:

Output Attribute	Value
clipper_TILE_NAME	Overview

6. Set KML Properties

When we view the dataset in Google Earth, we will want to be able to filter by each dataset (feature type). We can do this using the fme_feature_type attribute.

Add a KMLPropertySetter to the canvas. Connect both the AttributeCreator_3 (coming from the BoundsExtractor) to the KMLPropertySetter Input port. Then connect the Clipper Inside output port to the KMLPropertySetter Input port and the Inside output port on the Clipper.

Open the parameters and set the following:

Name: fme_feature_type
Visible: Yes

7. Filter Feature Types and Set Region

We want to set the region for each feature type. In this example, we will only set the region for a couple of feature types, for demonstration purposes.

Add another FeatureTypeFilter to the canvas and connect it to the KMLPropertySetter Output port. In the parameters dialog, add the following filter:

Feature Type Names:
- Buildings
- WaterMeters

This will filter for any Building and WaterMeters feature types.

8. Set Region for Buildings and WaterMeters

Add another KMLRegionSetter to the canvas. Connect both of the created FeatureTyleFilter_2 output ports (Buildings and WaterMeters) to the KMLRegionSetter_3 Input port.

Open the parameters and set the following:

Calculate from Feature Bounding Box: No
Minimum Display Size (pixels): 2048
Minimum X (West): _xmin
Minimum Y (South): _ymin
Maximum X (East): _xmax
Maximum Y (North): _ymax

9. Styling Schema with SchemaMapper Actions: Filter Features

To finish styling the Buildings and WaterMeters, as well as the other features, we will use a predefined schema and style using actions in a SchemaMapper. We will use three actions: Filter Features, Feature Type Map, and Set New Attributes. In this step, we will add the Filter Feature action.

Add a SchemaMapper to the canvas. Connect the KMLRegionSetter_3 Output port and the FeatureTypeFilter_2 <Unfiltered> output port to the SchemaMapper Input port:

In the SchemaMapper parameters, add the schema mapping table then click Continue:

Format: CSV (Comma Separated Value)
Dataset: /SchemaMapping.csv
- Click the file button to browse to the file location

In the SchemaMapper parameters, set the Format to Comma Separated Value (CSV), then browse to the SchemaMapping.csv file. Click Next.

Now we will create a series of Actions to define the schema rules. For more information see the SchemaMapper tutorial.

First, create a Filter action by clicking on the plus (+) sign at the bottom. From the drop-down menu, select Filter… and a Create Filter dialog will appear. Enter the following fields:

Attribute Name Field: SourceFeatureTypeAttribute
Attribute Value Field: SourceFeature

The selected fields will be highlighted in the Schema Mapping Preview.

Click OK. Click the plus (+) button again and create a second Filter action. Enter these attribute fields:

Attribute Name Field: FilterAttribute
Attribute Value Field: FilterAttributeValue

Click OK. Don’t close the SchemaMapper parameters just yet. We will add more actions in the next two steps.

10. Styling Schema with SchemaMapper Actions: Feature Type Map

Click the Feature Type Map… in the plus sign (+) drop-down menu from the SchemaMapper parameters, as done in the previous step.

Enter the following fields in the Create Feature Type Map dialog:

Source Feature Type Field: SourceAttribute
Destination Feature Type Field: KMLAttribute

Click OK and leave the parameters dialog open.

11. Styling Schema with SchemaMapper Actions: New Attribute

Finally, add a New Attribute action and set the following fields:

Attribute Name Field: FormatAttribute
Attribute Value Field: FormatAttributeValue

Click Done to save the SchemaMapper parameters.

12. Write Out Tiles

To write out our data analysis to KML, add another Writer from the top Toolbar. Set the following parameters in the Add Writer dialog:

Format: OGC / Google KML
Dataset: /Output.kml
- This will change when we set feature type fanout as we did previously
Feature Type Definition: Dynamic (Advanced)
- Dynamic feature type definition automatically changed the Feature Type Name to fme_feature_type

Click OK and connect the SchemaMapper Mapped output port to the <Dynamic> writer we just created:

13. Link User Parameter

We will, again, set the output destination to the User Parameter we created in Part 1.

In the Navigator pane, expand the second Output [OGCKML] writer. Right-click the Destination Google KML File and select Link to a Parameter…

In the Link to a Parameter dialog, link the output to the User Parameter we created, KMLDIR:

Link to: KMLDIR

14. Set Up Dataset Fanout

We want the <Dynamic> writer to fanout the dataset using the clipper_TILE_NAME attribute we created.

In the Navigator, under the second OutputKML (OutputKML [OGCKML] - 2), double-click the Fanout Dataset option.

In the Edit Output [OGCKML] -2 dialog, set the parameters as follows:

Fanout Dataset: Enabled
Fanout Expression:
```
@Value(_networkLinkFile).kml
```

14. Run Workspace and View Output

Run the workspace using the green Run button in the Toolbar. For the Feature Types to Read parameter, click the ellipsis to select all feature types:

Once the workspace is run, open the output folder. This can be done by selecting the writer feature type on the canvas and clicking the folder icon . There will be three different types of files:

KML_Network_Link.kml
- All the tiles
TILE_Overview.kmz
- Overview tile for the area of interest
TILE_<tile name>.kmz
- KML regions (tiles) with features

Open the KML_Network_Link.kml dataset in Google Earth. You may have to zoom in slightly to see the data as we set a minimum pixel display size.

If you double-click on a tile in the Places pane, it will zoom in and show more detail. The tile below is showing the detail for L27.