FME and Metadata

FME & Metadata Overview

In a world overflowing with data metadata is increasingly important to allow us to find and properly manage it. Support for standards such as OGC and ISO provide us a consistent means for working with metadata. FME supports metadata processing, but to a large extent how you go about it depends on the format. In some cases, formats have special metadata records that FME's reader/writer is able to detect, read and modify. An example of this is Geodatabase metadata which we support reading from and writing to. In other cases, metadata exists as a separate file such as ISO 19115 XML or JSON that accompanies and describes a SHAPE or GML file. In this case FME supports reading and writing this XML or JSON with the appropriate XML or JSON reader/writer. In either case, FME also has a comprehensive range of transformers that lets you create, and modify metadata either as metadata objects (XML / JSON) or as individual metadata attributes, depending on how you configure your reader / writer.

Metadata may occur at the dataset level, such as with geodatabase mentioned above. In other cases there are metadata attributes on every feature. This is often the case with raster formats such as GeoTIFF, jpeg, netCDF etc. We expose many fields that other tools may miss and give you full control over the schema and content. One common use case for FME is to support automated updates to dataset metadata for things like data extents, coordinate system, value ranges, last modified date, contact info etc.

In the case of shape files there is a convention whereby ISO 19115, FGDC or other standard metadata can be stored in a <filename>.xml file to go along with the <filename>.shp file to give a dataset level metadata record. Again, you are free to read this with either the XML or XSD XML reader. You can then use xml transformers such as the XQueryExtractor, XMLUpdater and XMLTemplater to modify the xml properties for things like update time and author etc.
 

Two FME Approaches for Working with Metadata

Basically there are 2 approaches for working with this type of XML metadata: templates and application schema based metadata features. The template approach involves taking a standard form of the metadata, such as an existing xml template that you are using for your organization, and then using FME to modify or insert new xml records based on the template. For inserting new records, FME can be configured to reading the metadata template, and replace the fields or properties relevant to any required new metadata record, based upon the author, time, and other relevant properties of the data being processed. For updates or migration of existing records, FME can simply be configured to read and write metadata as is, and only modify the properties relevant to the update. If the source is compliant with the standard you are using, the destination will be also. This typically involves using the XML Reader or XMLFragmenter to read or consume the XML metadata, then a range of XML transformers to parse and update the data, and finally one or more XMLTemplaters to compose your XML metadata document before writing it out with a Text File writer.

The other approach is to read the metadata as FME records with metadata attributes. Recently we added the XSD XML Reader / Writer which can be configured to read / write metadata XML directly using the ISO 19115 XML application schemas. This allows FME to read the metadata schema directly and create feature type definitions for the various metadata components such as MD_DataIdentification, EX_Extent, DQ_DataQuality CI_ResponsibleParty, etc. The key with the XSD XML Reader is to use the tree control to select the appropriate nodes of the XML document structure to interpret as FME features based on what you need to read and / or modify. At this point there is not a specific ISO 19115 reader, so there is some understanding of the 19115 structure and associated reader configuration required. The advantage of this approach is that once you have read the metadata elements as FME features, its a lot easier to work with the metadata properties as FME features and attributes using typical string transformers such as AttributeCreator etc.

In short, in terms of compliance with FGDC and ISO 19115, 19117 standards, while we do not supply templates 'out of the box', it is possible to comply with these standards using FME, wherever your metadata may be stored or consumed.
 

Metadata Articles

For more information on working with metadata see the following articles:

Working with Geodatabase Metadata: Writing/Updating Metadata

Harvesting and writing ISO19115 XML Metadata

Using the XMLUpdater transformer to manipulate Metadata strings

Reading LAS Metadata with FME

Metadata Validation

Tutorial: Getting Started with XML

Metadata Webinars

Also, it would be worthwhile to review the following webinars which all have some discussion and examples related to metadata:

Metadata Matters! What it is and How to Manage it

Mastering Geospatial Metadata with FME

XMLathon with Don and Dean

Reading XML

Writing XML