Dynamic Workflows: Destination Schema Derived from Multiple Sources

Introduction

In a dynamic workspace, the Writer schema is obtained from either the schema of an existing dataset or a schema constructed with List Attributes. The Writer schema is usually obtained from a single source, but sometimes a single Writer schema needs to be constructed from multiple sources.

This example constructs the Writer schema using List Attributes and multiple sources to obtain the required information. This technique merges multiple datasets and writes them out with a combined schema, all dynamically.

The workspace in this example translates a dataset of parks. Attributes from a secondary dataset need to be added to the parks and written out. Because these attributes are not known in advance, a dynamic translation is necessary, and because there are multiple schema sources, a List Attributes technique must be used.

Workspace and Data

The workspace referenced in this example can be found in the zip file attached to this tutorial, titled "6-MultipleSources". The data used here is adapted from data made available by the City of Vancouver, British Columbia.

Creating the List

As noted in the previous article on List Attributes, the List Attributes required to construct a dynamic schema are attribute{}.name and attribute{}.fme_data_type. These define the names and data types of the Writer schema attributes. They can either be constructed from scratch or extracted from the main dataset using a Reader format called "Schema (Any Format)". For example:

Similarly, a list of attributes can be extracted from the secondary data source:

The main problem the workspace must solve is merging these two lists into a single output schema definition.

The Workspace

The workspace looks like this:

There are three readers:

• Primary Dataset Reader
• Secondary Dataset Reader
• Schema Reader

The Schema Reader reads the schema from both source datasets (primary and secondary) and so emits two features. The attribute lists are exploded to create one feature per attribute and then merged back into a single list using an Aggregator transformer. Excess schema information is removed with an AttributeKeeper transformer:

The primary and secondary datasets are read – the primary dataset here is fixed, but the secondary dataset is generic/dynamic – and the information is transferred from secondary to primary (here using a SpatialFilter):

Finally, the schema list attributes are merged onto the primary features using the FeatureMerger transformer:

The Writer Feature Type is configured dynamically to use the Schema Feature as its schema source.

Output

Run the workspace and inspect the output. You will find that each park now has information attached indicating which development zone it is in.

The important part is that you can change the source zoning dataset to something else (of any format), and the attributes from those features will be dynamically attached and used in the output.

Notes

• Here, the primary dataset reader is set to a fixed schema and the MapInfo format, but there is no need for that to be so. Similarly, the writer could be generic enough to support any data format.
• The secondary dataset can be any format because the reader is Generic.
• The secondary dataset can use any schema, but it must contain features that overlap with the parks in the SpatialFilter.
• The source dataset parameter for the Schema Reader shares parameters with both the primary and secondary datasets, so the same dataset doesn't need to be selected twice.

Step-by-step Instructions

1. Add Readers and Schema

Add two MITAB Readers, one for Zones data and one for Parks data, and ensure that Single Merged Feature Type is selected for both.

• Format: MapInfo TAB (MITAB)
• Dataset: <Path>\Zoning\Zones.tab
• Workflow Options: 
  • Single Merged Feature Type: enabled

• Reader Format: MapInfo TAB (MITAB)
• Reader Dataset: <Path>\Parks\Parks.tab
• Workflow Options: 
  • Single Merged Feature Type: enabled

Also, add a Schema Reader. Add the Zones.tab file first and then click Parameters. In the Additional Attributes to Expose parameters, add the following list. Click OK to add the Reader to the workspace. In the Navigator window, maximize Zones [SCHEMA] and click on the Source Dataset. Click the dropdown, then select Select Multiple Files/Folders. Add the Parks.tab file.

• Format: Schema (Any Format)
• Dataset: <Path>\Zoning\Zones.tab, <Tutorial>\Parks\Parks.tab

• Parameters: 

  • Additional Attributes to Expose:
    • attribute{}.fme_data_type
    • attribute{}.name
    • attribute{}.native_data_type
    • fme_basename
    • fme_format_long_name
    • fme_format_short_name

  Click OK

2. Set up SpatialFilter

Connect a SpatialFilter to the Zones Writer via the Filter Input port and the Parks Writer via the Candidate Input port. In the parameters set the Spatial Predicates to Test:

• Filter Contains Candidate
• Filter Intersects Candidate
• Filter is Within Candidate
• Filter Touches Candidate

3. ListExploder

Add a ListExploder to the Schema Writer. In the parameters set, List Attribute to attribute{}.

4. Aggregator

Connect an Aggregator to the Elements Output port on the ListExploder. Set the Accumulation Mode to Merge Incoming Attributes. Check Generate List, set List Name to attribute, and for Selected Attributes add fme_data_type and name

5. AttributeKeeper

We are only interested in keeping two of the attributes that we exposed. Add an AttributeKeeper after the Aggregator. In the parameters, select the … in Lists to Keep and select attribute{}.fme_data_type and attribute{}.name

6. FeatureMerger

To merge everything, we will use the FeatureMerger. Connect both the Passed and Failed Output ports on the SpatialFilter to the Requestor Input port and the AttributeKeepers Output port to the Supplier Input port on the FeatureMerger. In the FeatureMerger parameters, set the Requestor to 3 and the Supplier to 3, keep everything else as the defaults.

7. Write out to a dynamic writer

Add an MITAB Writer to the workspace and ensure it is set to Dynamic (Advanced). Connect it to the Merged Output port on the FeatureMerger. Once added to the workspace, in the parameters, set the Schema Sources to “Schema From Schema Feature”. Connect an Inspector to the Merged Output port and run translation.

• Format: MapInfo TAB (MITAB)
• Dataset: <Tutorial>\Output
• Table Definition: Dynamic (Advanced)

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.