Dynamic Workflows: Destination Schema Derived from Multiple Sources

Introduction

In a dynamic workspace, the Writer schema is obtained from either the schema of an existing dataset or a schema constructed with List Attributes. The Writer schema is usually obtained from a single source, but sometimes a single Writer schema needs to be constructed from multiple sources.

This example constructs the Writer schema using List Attributes and uses multiple sources to obtain the information needed. This technique merges different datasets and writes them out with a combined schema, all in a dynamic way.

The workspace in this example translates a dataset of parks. Attributes from a secondary dataset need to be added to the parks and written out. Because these attributes are not known in advance, it is necessary to use a dynamic translation, and because there are multiple schema sources, a technique that uses List Attributes must be used.
 

Workspace and Data

The workspace referenced in this example can be found in the zip file attached to this tutorial, titled "6-MultipleSources". The data used here is adapted from data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.
 

Creating the List

As noted in the previous article on List Attributes, the List Attributes required to construct a dynamic schema are attribute{}.name and attribute{}.fme_data_type. These define the name and data type for the Writer schema attributes. They can either be constructed from scratch or extracted from the main dataset using a Reader format called "Schema (Any Format)". For example:
 

Similarly, a list of attributes can be extracted from the secondary data source:
 

The main problem to be solved by the workspace is to merge these two lists into a single output schema definition.
 

The Workspace

The workspace looks like this:

There are three readers:

• Primary Dataset Reader
• Secondary Dataset Reader
• Schema Reader

The Schema Reader reads the schema from both source datasets (primary and secondary) and so emits two features. The attribute lists are exploded to create one feature per attribute and then merged back into a single list using an Aggregator transformer. Excess schema information is removed with an AttributeKeeper transformer:
 

The primary and secondary datasets are read – the primary dataset here is fixed, but the secondary dataset is generic/dynamic – and the information is transferred from secondary to primary (here using a SpatialFilter):
 

Finally, the schema list attributes are merged onto the primary features using the FeatureMerger transformer:
 

The Writer Feature Type is set up dynamically and to use the Schema Feature as a schema source.
 

Output

Run the workspace and inspect the output. You will find that each park now has information attached about what development zone the park resides in.

The important part is that you can change the source zoning dataset to something else (of any format) and the attributes from those features will be dynamically attached and used in the output.

Notes

• Here the primary dataset reader is fixed schema and set to MapInfo format, but there is no need for that to be so. Similarly, the writer could be generic to enable any data format to be written.
• The secondary dataset can be any format because the reader is Generic.
• The secondary dataset can be any schema, but it needs to contain features that overlap the parks in the SpatialFilter.
• The source dataset parameter for the Schema Reader shares the parameters for both the primary and secondary datasets, meaning the same dataset doesn't need to be selected twice.

Step-by-step Instructions

1. Add Readers and Schema

Add two MITAB Readers, one for Zones data and one for Parks data, ensure that Single Merged Feature Type is selected for both.
 

• Format: MapInfo TAB (MITAB)
• Dataset: <Tutorial>\Zoning\Zones.tab
• Workflow Options: Single Merged Feature Type

• Reader Format: MapInfo TAB (MITAB)
• Reader Dataset: <Tutorial>\Parks\Parks.tab
• Workflow Options: Single Merged Feature Type

Also add a Schema Reader. Add the Zones.tab file first and then click Parameters. In the parameters for Additional Attributes to Expose, add the following list. Click OK to add the Reader to the workspace. In the Navigator window, maximize Zones [SCHEMA] and click on the Source Dataset. Click on the dropdown, and click Select Multiple Files/Folders. Add the Parks.tab file.

• Format: Schema (Any Format)
• Dataset: <Tutorial>\Zoning\Zones.tab, <Tutorial>\Parks\Parks.tab
• Parameters: Additional Attributes to Expose:
  • attribute{}.fme_data_type
  • attribute{}.name
  • attribute{}.native_data_type
  • fme_basename
  • fme_format_long_name
  • fme_format_short_name

2. Set up SpatialFilter

Connect a SpatialFilter to the Zones Writer via the Filter Input port and the Parks Writer via the Candidate Input port. In the parameters set the Spatial Predicates to Test:

• Filter Contains Candidate
• Filter Intersects Candidate
• Filter is Within Candidate
• Filter Touches Candidate

3. ListExploder

Add a ListExploder to the Schema Writer. In the parameters set List Attribute to attribute{}.

4. Aggregator

Connect an Aggregator to the Elements Output port on the ListExploder. Set the Accumulation Mode to Merge Incoming Attributes. Check Generate List, set List Name to attribute and for Selected Attributes add fme_data_type and name

Note: Older versions of FME won't have the Add To List or Select Attributes parameters.

5. AttributeKeeper

We are only interested in keeping two of the attributes that we exposed. Add an AttributeKeeper after the Aggregator. In the parameters select the … in Lists to Keep and select attribute{}.fme_data_type and attribute{}.name

6. FeatureMerger

To merge everything, we will use the FeatureMerger. Connect both the Passed and Failed Output ports on the SpatialFilter to the Requestor Input port and the AttributeKeepers Output port to the Supplier Input port on the FeatureMerger. In the FeatureMerger parameters, set the Requestor to 3 and the Supplier to 3, keep everything else as the defaults.

7. Write out to a dynamic writer

Add a MITAB Writer to the workspace, ensure that it is set to Dynamic (Advanced). Connect it to the Merged Output port on the FeatureMerger. Once added to the workspace, in the parameters set the Schema Sources to “Schema From Schema Feature”. Connect an Inspector to the Merged Output port and run translation.

• Format: MapInfo TAB (MITAB)
• Dataset: <Tutorial>\Output
• Table Definition: Dynamic (Advanced)