Dynamic Workflows: Destination Schema as a Mirror Image of the Source Dataset

Introduction

A dynamic workflow that mirrors the source and destination schema is simple to set up and offers maximum flexibility. The user can input any dataset of the chosen source format, and the workflow will create an output that is a mirror of the input dataset’s schema. For example, a workflow can be created so that any table from a geodatabase can be read and converted to a CityGML file while maintaining its schema.

The important part is that a user can change the source to a different dataset and the translation will still work. The user will not need to make changes to the workspace. This approach is perfect for when a dataset has many feature types with different schemas, or when new feature types will be added to the workflow in the future.

In the following example, an OGC Geopackage dataset (CityDatabase.gpkg) contains multiple feature types, each with a different schema. Each feature type will be reprojected and written dynamically to shapefile using its own schema.
 

Workspace and Data

A finished version of the workspace created in this example can be found in the zip file attached to this tutorial, titled "1-MirrorImage".

CityDatabase.gpkg contains several tables (feature types), which can be seen in the Select Tables dialog:

Step-by-Step Instructions

1. Open FME and Select the Generate Workspace Tool

Open FME Workbench and select the Generate Workspace tool from the start page. We will configure this tool over the next three steps. 

Configure the reader as follows:

• Format: OGC Geopackage
• Dataset: <Tutorial Download>\resources\CityDatabase.gpkg
• Parameters > Tables: BusStops, CityParks, Roads

Configure the writer as follows:

• Format: Esri Shapefile
• Dataset: <Tutorial Download>\output 

2. Set the Workspace to be Dynamic

In the dialog, there are two Workflow Options: Static Schema and Dynamic Schema. If the Static Schema option is selected, each feature type would appear separately on the Workbench canvas like this:


However, if the Dynamic Schema option is chosen, a schema-less workspace with both a dynamic reader and writer is created.

Select 'Dynamic Schema'.

3. Click OK to Generate the Workspace
Click OK to generate the workspace. Both the dynamic reader and writer will have a single feature type displayed on the canvas, regardless of the number of reader feature types selected.


The reader feature type is named ‘<All>’ while the writer feature type is named ‘<Dynamic>’. As well, the dynamic feature type does not contain any attributes.

Behind the scenes, the reader feature type has been configured to read all features regardless of the feature type name by setting the Merge Filter to an asterisk (*). This can be seen by opening the reader feature type parameters.



The writer feature type has been configured to write all features with the schema defined by the source dataset.

Note: Individual readers and writers can be made dynamic without having to use the Generate Workspace dialog. This is done by setting the options above in each feature type’s parameters dialog.
 

4. Add a Reprojector Transformer
Click anywhere on the canvas and begin typing 'Reprojector'. Place this transformer between the reader and writer feature types.

Double-click the Reprojector to open the parameters, and set the Destination Coordinate System to LL84.
 

5. Run the Workspace

When the workspace is run, all of the CityDatabase source data is read through the single feature type.

On the writer side, the incoming “fme_feature_type” attribute contains the feature type name and is used to determine which schema to use. For example, a feature with “fme_feature_type” as “Roads” will search the CityDatabase for the “Roads” dataset, and when found, it will use that schema. The data will then be dynamically divided back into its component layers, using the appropriate schema to organize the original attributes and geometry type.
 

Advanced Techniques

Using the Feature Types to Read Parameter

When a dynamic workflow is created, an extra User Parameter called “Feature Types to Read” is published automatically within the workspace.

This parameter creates a quick way to change the number of feature types to be read. When the workspace is run using “Prompt for User Parameters”, the user can select which feature types to read in the dialog that appears.

This is especially useful if the workspace references dozens of source feature types (for example, a database) and not all of them are required during a dynamic translation run.
 

Adding Additional Attributes to a Dynamic Writer

Since the destination feature type is a mirror of the source dataset’s schema, attributes created within the workspace are not automatically added to the output. Any new attributes need to be added within the User Attributes tab in the writer feature type parameters. For example, the "_timestamp" attribute created by the DateTimeStamper transformer has been manually added to the writer feature type here:

And it appears on the writer feature type:

SHAPEFILE Writer requires attribute names to start with a letter, which includes both Latin and non-Latin alphabets (such as Japanese, Cyrillic, or Arabic scripts). If an attribute name begins with a non-letter character (such as numbers or special characters), the Writer will automatically add a 'Q_' prefix to ensure compatibility.

Generic Workspace

Adding a Generic Reader and/or Generic Writer to this workflow could produce one single workspace capable of reading/writing any format of data, regardless of what schema it has.
 

Additional Resources

• Generic Readers/Writers

Data Attribution

The data used here is adapted from data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.