Integration example for OGC CSW Data Catalog And Repository with FME Server

Introduction

This example demonstrates how FME Server can be integrated with OGC Catalog Service for the Web (CSW) technology to build an OGC-based Metadata Data Repository. The demonstration system provides data upload, data browsing, and data download capabilities as described below.

First, users are able to upload any spatial data supported by FME to the system. This can be GIS, CAD, Raster, or 3D data. When data is published to the example system it also creates and publishes metadata information to an OGC Catalog Service for the Web(CSW).

Second, users are able to browse the OGC CSW meta-data using Google Earth. A user browses the catalog by simply navigating to an area of interest. If there is data there then data footprints are shown. To see this in action click on the link below and zoom into the area of Austin, Texas. When you get to the appropriate zoom level you will see the data footprints appear.

Third, users are able download the data in the original form or in a different format. This is done if a user clicks on one of the data footprints. Upon clicking they are presented with a pop-up box that contains information about the particular data-set that they clicked on. Users can then download the data in either its original form or in another format right from the pop-up.

The scenario thus presents an open source standards based data sharing system that encapsulates data publishing, data searching, and data extraction. As each aspect is controlled by a single workspace this can form the basis for more complex systems.

As this is a demonstration system Safe makes no guarantee on any aspect of this demonstration system or the data that is uploaded or downloaded. It should also be pointed out that any data posted to the site is available to all other users of the site. This site is purely for demonstration purposes.

The example encompasses the following capabilities outlined below.

Data Publishing

Thru a simple web interface users are able to upload any data types that FME can read. Simple identify the file or files that make up your dataset or optionally a *.zip file with the entire dataset. So for example if you are loading a Mapinfo MIF/MID dataset you can do it by individually uploading the MIF and MID files or you can zip them together and upload the zip file.

Data Searching

Here we take advantage of the KML Network Link capabilities of FME Server. As a user zooms in we use the bounding box of the zoom area to query the OGC CSW for data in the area. The query at this point is simple but could be easily expanded to exploit more search capabilities of the OGC CSW.

Data Download

From within Google Earth users simply click on a data footprint and are given some information about the data and the ability to download the data in either its original form or a limited set of formats.

Bulk Loading

There is also available a sample workspace that shows how you can use an FME Desktop application to post a large amount of data to the site. So for example if you have a large number of shape files that have the same user entered metadata then you can run this workspace and it will upload them to the server using the web services capabilities of the FME Server

How It Works

The system consists of 4 workspaces that were created using FME Desktop (Workbench), published to FME Server and registered with FME Server services. The details of the workspaces and the services that they are using as well as other notable FME technology is described below.

Publishing of Data to Repository and Catalog (D010-PublishData.fmw)

The first workspace takes the user specified meta-data information along with the uploaded data. It extracts additional meta-data information from the input file and combines this with the meta-data information that is passed in thru its parameters. The workspace first extracts additional meta-data from the input file. The meta-data information includes file type, file name, coordinate system, and bounding box. The metadata also includes a URL which has the location of the file. The point of the OGC catalog is that many different applications could use the catalog and so it is important that the catalog entry tell the user how to get the data. The workspace then uses the XMLTemplater transformer to build the "CSW Insert" statement to populate the OGC meta-data catalog. The workspace also moves the input data from the web server upload location to a permanent holding location using the FileCopy writer.

At the conclusion of this operation the meta-data record is created and inserted as a new Catalog record into the OGC CSW and the uploaded data is put the data into a permanent location.

Browsing OGC CSW metadata catalogue (D010-StreamCSWContents.fmw)

This workspace is registered with the KML Network Link service of FME Server. It is registered in such away that a request from Google Earth is only triggered when the user zooms in to a certain scale.

This workspace is passed in the Bounding Box of the area that is being viewed by Google Earth. The workspace uses this to build a bounding box query against the OGC CSW. This request is again built using the XMLTemplater workspace. For this demo (in the interest of time) the OGC CSW only returns the first 10 records that match the bounding box query and those are parsed using XQuery transformers and returned as KML bounding boxes to Google Earth.

The workspace also embeds in the KML document a pop-up HTML box that enables the user to invoke four different data download activities. One is to get the original data and meta-data, the second is to request the data in a different format. The final two are to stream the data into Google Earth so that you can inspect the data and finally you are able to download the metadata record from the OGC Catalogue service. This popup was again generated using the XMLTemplater.

Meta Data Download (D010-MetaDataDelivery.fmw) This workspace is registered with the data download service and simply extracts the metadata record and returns it to the caller as an XML document.

Original Data Download (D010-OriginalDataDelivery.fmw) This workspace is registered with the data download service and is very simple. It is given the meta data id and using this it extracts the meta-data record from the CSW and delivers this to the user along with the original data. The package is delivered in a compressed file.

Translated and download Data (D010-TranslateDataDelivery.fmw) This workspace is registered with the data download service and is the simplest of them all. It simply uses the generic reader and writer to change the format of the dataset to that specified. Dynamic translation is also engaged.

Stream Data as KMZ (D010-DataStreamAskmz.fmw) This workspace renders the uploaded data right in google earth so that the dataset of interest can be viewed. It delivers both geometry and attribution to the end user. The workspace is setup to process only the first 3000 features in order to keep the datasize returned by the streaming small to get reasonable performance.

Setting up the Example

To actually set up and run the example on your own system requires a certain set of software and setup.

Requirements

• Google Earth
• PostgreSQL
• OGC CSW (We are running Deegree here but also have it working on others in our office.)
• FME Server 2010+
• FME Desktop 2010+
• workspaces and required files

Note: We have tested the attached package using Deegree version 3.1.2 and PostgreSQL 8.3 with PostGIS 1.3.6

Steps

• Download the workspaces and required files zip file.
• Extract the zip file into C:\FMEDemos.
• Open the following readme file and follow the directions inside:
• C:\FMEDemos\D010-OGC Data Catalog and Repository\Documentation\readme.txt