Dynamic Workflows: Destination Schema is Derived from a Schema Feature

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2022.0

Introduction

This type of workflow dynamically creates the output schema using list attributes. In FME, list attributes allow a single attribute to store multiple values. Dynamic writers check the first feature of each incoming feature type for the presence of a specific list attribute, which it then uses to generate the output schema.

In this example, a workspace translates city park datasets to the MapInfo TAB format with a new schema. This schema should include park name information and remove everything else. In a standard dynamic workflow, like in the previous examples in this tutorial series, the output schema would mirror the input dataset, which is not what is needed in this case.
 

Workspace and Data

The workspace referenced in this example can be found in the zip file attached to this tutorial, titled "5-ListAttributes".
 

How to Create a Schema List Attribute

The AttributePivoter transformer, SchemaSetter transformer, and the “Schema (any format)” Reader can all create a schema list. The FeatureReader transformer can also generate an attribute list using a ‘Schema’ output port.

A simple attribute list looks something like this:

Attribuets.png

In this example, each attribute{} within the list has two components:

  • name: the name of the attribute to be included in the schema.
  • fme_data_type: the data type of the attribute.


Valid FME data types include:

Type Fields
Character fme_varchar(width), fme_char(width), fme_char
Integer fme_uint8, fme_int16, fme_uint16, fme_int32, fme_uint32, fme_int64, fme_uint64
Numeric fme_decimal(width,decimal), fme_real32, fme_real64
Date-Time fme_datetime, fme_time, fme_date
Other fme_buffer, fme_boolean


Using Format-Specific Data Types

If you wish to use format-specific data types (i.e. non-FME data types), include two more attributes:

Attribute Description
fme_format_short_name Format for which that type belongs, using the short format name.
attribute{0}.native_data_type Format specific data type.

Example:

fme_format_short_name = “GEODATABASE_FILE”
attribute{0}.name = "myAttribute"
attribute{0}.fme_data_type = "fme_buffer"
attribute{0}.native_data_type = "guid"

The outcome will be a new geodatabase feature class containing an attribute ‘myAttribute’ of data type “guid”.
 

Adding Geometry

For formats that require a specific geometry for each feature type (e.g. shapefile, geodatabase), you need to define the geometry type. This is done using the fme_geometry{} list attribute.

Attribute Description

fme_geometry{}

Format for which geometry belongs, you can define more than one geometry if needed.

For example:

fme_geometry{0}.fme_point 

This would define a point geometry feature class for a Geodatabase.

Note: When generating a list, as shown above, you should merge it onto your data so that the writer gets both the schema and the data when the first feature enters it.

 

Step-by-step Instructions

1. Add a Reader to a blank workspace
Open a blank workspace in FME Workbench. Add a reader with the following parameters:

  • Format: OGC Geopackage
  • Dataset: <Tutorial Download>\resources\CityDatabase.gpkg
  • Parameters > Tables: CityParks

Geopkg.png
 

2. Set up Schema Mapping

Add an AttributeManager to the canvas and connect it to the CityParks reader feature type. Open the parameters and change the names of the three attributes to match the database schema.

  • PRIMARYINDEX -> PID
  • name -> PARK_NAME
  • name_alt -> ALT_NAME
  • id (remove)

AttributeMan.png

3. Define a New Schema

Add an AttributeCreator after the AttributeManager. In the parameters, define the new schema as follows:

Attribute Name Value
attribute{0}.name PARK_NAME
attribute{0}.fme_data_type fme_varchar(20)
attribute{1}.name ALT_NAME
attribute{1}.fme_data_type fme_varchar(20)

AttributeCreator.png

 

4. Add a Dynamic Writer

Add a writer to the workspace

  • Format: MapInfo TAB (MITAB)
  • Dataset: <Tutorial Download>\output
  • Table Definition: Dynamic (Advanced)

Once the writer is added to the workspace, connect it to the AttributeCreator and open up the writer parameters. Change the Schema Source to be "Schema From Schema Feature".

mitabwriter.png


5. Run the Workspace and Inspect the Results

Here is the final workspace. Run the workspace and inspect the results. 
image

Advanced Examples

Using the FeatureReader

The FeatureReader transformer can be used to read data into a workspace and perform a spatial query on it. The output schema is specified using list attributes generated by the FeatureReader and passed through the schema output port. In these scenarios, list attributes are the best option is to define your schema, as this information is not known until the workspace is run.

See this article for a full example: Extracting data within a clipping polygon
 

Modifying the Schema Feature

See Modifying the Schema Feature for a deeper look at the schema feature and how to edit it within a workspace.

 

Notes

Geometry Error

If there is a "Geometry Not Specified" error when using this technique, try setting the attribute: fme_geometry{0} = <fme_type>. This applies to shapefiles and some other formats. For more on <fme_type> see here: FME Geometry Types

 

Avoiding Dropped Features - FeatureReader

If using the FeatureReader to create a schema feature, note the transformer adds a special attribute and value:

fme_schema_handling = ‘schema_only’

This attribute tells the writer to use that feature as a schema and then remove it from the output. This can be an issue if you do any merging of data and schema in the workspace, as the data merged with this schema feature would be dropped when written. For instance, if you merge a schema feature so it gets added to all the features in the workspace.

To ensure this doesn’t occur, remove the fme_schema_handling attribute entirely so the first merged data feature is not dropped by the Writer.


Additional Resources

 

Data Attribution 

The data used here is adapted from data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.