Dynamic Workflows: Destination Schema is Derived from a Schema Feature

Introduction

This type of workflow dynamically creates the output schema using list attributes. In FME, list attributes allow a single attribute to store multiple values. Dynamic writers check the first feature of each incoming feature type for the presence of a specific list attribute, which they then use to generate the output schema.

In this example, a workspace translates city park datasets to the MapInfo TAB format with a new schema. This schema should include park name information and remove everything else. In a standard dynamic workflow, as in the previous examples in this tutorial series, the output schema would mirror the input dataset, which is not needed in this case.

Workspace and Data

The workspace referenced in this example can be found in the zip file attached to this tutorial, titled "5-ListAttributes".

How to Create a Schema List Attribute

The AttributePivoter transformer, SchemaSetter transformer, and the “Schema (any format)” Reader can all create a schema list. The FeatureReader transformer can also generate an attribute list using a ‘Schema’ output port.

A simple attribute list looks something like this:

In this example, each attribute{} within the list has two list elements:

• name: the name of the attribute to be included in the schema.
• fme_data_type: the data type of the attribute.

Valid FME data types include:

Using Format-Specific Data Types

If you wish to use format-specific data types (i.e., non-FME data types), include two more attributes:

Example:

fme_format_short_name = “GEODATABASE_FILE”
attribute{0}.name = "myAttribute"
attribute{0}.fme_data_type = "fme_buffer"
attribute{0}.native_data_type = "guid"

The outcome will be a new geodatabase feature class containing an attribute ‘myAttribute’ of data type “guid”.

Adding Geometry

For formats that require a specific geometry for each feature type (e.g., shapefile, geodatabase), you need to define the geometry type. This is done using the fme_geometry{} list attribute.

For example:

fme_geometry{0}.fme_point 

This would define a point geometry feature class for a Geodatabase.

Note: When generating a list, as shown above, merge it into your data so the writer receives both the schema and the data when the first feature enters it.

Step-by-step Instructions

1. Add a Reader to a blank workspace
Open a blank workspace in FME Workbench. Add a reader with the following parameters:

• Format: OGC Geopackage
• Dataset: <Path>\resources\CityDatabase.gpkg
• Parameters:
  • Tables: CityParks

2. Set up Schema Mapping

Add an AttributeManager to the canvas and connect it to the CityParks reader feature type. Open the parameters and rename the three attributes to match the database schema.

• Input Attribute: PRIMARYINDEX 
  • Output Attribute: PID
• Input Attribute: name 
  • Output Attribute: PARK_NAME
• Input Attribute: name_alt 
  • Output Attribute: ALT_NAME

Remove the following attribute:

• Input Attribute: id
  • Action: Remove

Click OK

3. Define a New Schema

Add an AttributeCreator after the AttributeManager. In the parameters, define the new schema as follows:

• Attribute Name: attribute{0}.name
  • Attribute Value: PARK_NAME
• Attribute Name: attribute{0}.fme_data_type
  • Attribute Value: fme_varchar(20)
• Attribute Name: attribute{1}.name
  • Attribute Value: ALT_NAME
• Attribute Name: attribute{1}.fme_data_type
  • Attribute Value: fme_varchar(20)

4. Add a Dynamic Writer

Add a writer to the workspace

• Format: MapInfo TAB (MITAB)
• Dataset: <Path>\output
• Table Definition: Dynamic (Advanced)

Once the writer is added to the workspace, connect it to the AttributeCreator and open up the writer parameters. Change the Schema Source to be "Schema From Schema Feature".

5. Run the Workspace and Inspect the Results

Here is the final workspace. Run the workspace and inspect the results. 

Advanced Examples

Using the FeatureReader

The FeatureReader transformer can be used to read data into a workspace and perform a spatial query on it. The output schema is specified using list attributes generated by the FeatureReader and passed through the schema output port. In these scenarios, list attributes are the best option is to define your schema, as this information is not known until the workspace is run.

See this article for a full example: Extracting data within a clipping polygon

Modifying the Schema Feature

See Modifying the Schema Feature for a deeper look at the schema feature and how to edit it within a workspace.

Notes

Geometry Error

If there is a "Geometry Not Specified" error when using this technique, try setting the attribute: fme_geometry{0} = <fme_type>. This applies to shapefiles and some other formats. For more on <fme_type> see here: FME Geometry Types

Avoiding Dropped Features - FeatureReader

If using the FeatureReader to create a schema feature, note that the transformer adds a special attribute and value:

fme_schema_handling = ‘schema_only’

This attribute tells the writer to use that feature as a schema and then remove it from the output. This can be an issue if you merge data and schema in the workspace, as the merged data with this schema feature would be dropped when written. For instance, if you merge a schema feature so it gets added to all the features in the workspace.

To ensure this doesn’t occur, remove the fme_schema_handling attribute entirely so the Writer doesn't drop the first merged data feature.

Additional Resources

• Example using the Schema (Any Format) Reader
• List Attributes

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.