Dynamic Workflows: Destination Schema is Derived from an External Dataset

Introduction

The workflow in the previous example reads any data and dynamically copies the schema from source to destination, generating an output that is a mirror of the input.

It is also possible to map the source data into a new schema and write that dynamically. This is useful to enforce a particular output or when the desired schema already exists. Any reader dataset in any format can be used as the template for the destination schema by adding it as a resource reader.
 

Workspace and Data

A finished version of the workspace created in this example can be found in the zip file attached to this tutorial, titled "2-ExternalDataset".
 

Step-by-Step Instructions

In this example, a project provides updated city data as a standardized set of shapefiles. However, updates are received in the OGC Geopackage format. To handle this scenario, a dynamic workspace is created using the standardized shapefiles as schema resources.

1. Add the Source Dataset

Open a new FME Workspace. From the Readers menu, select "Add Reader" and configure it as follows:

• Format: OGC Geopackage
• Source: <Tutorial Download>\resources\CityDatabase.gpkg
• Parameters > Tables: BusStops, CityParks, Roads
• Workflow Options: Individual Feature Types

This will read the three selected feature types in the dataset.


 

2. Add the Destination Dataset

From the Writers menu, select "Add Writer" and configure it as follows:

• Format: Esri Shapefile
• Dataset: <Tutorial Download>\output
• Shapefile Definition: Dynamic (Advanced)

3. Use Workspace Resources to set the Schema

In the Readers menu, select "Add Reader as Resource."


Configure it as follows:

• Format: Esri Shapefile
• Dataset: <Tutorial Download>\resources\schema_templates\BusStops.shp,<Tutorial Download>\resources\schema_templates\CityParks.shp,<Tutorial Download>\resources\schema_templates\Roads.shp (hold shift to select all three at the same time)

Once the resources are added, an entry is placed in the Navigator window:

 

4. Set the Dynamic Writer to use the New Schema

To map the source data into a new dynamic schema, open the Writer Feature Type parameters and use the “Schema Sources” setting. By default, the schema from the original source dataset, in this case 'CityDatabase', is entered into the dialog. Change the setting to point to the new schema:



Click OK and close the dialog.

5. Map the Schema

Finally, configure the workspace to map the source to the new schema. Add an AttributeManager to the canvas and connect it to all three reader feature types, then connect the output port to the writer. In the parameters, change the following:
Rename:

• PRIMARYINDEX -> PID
• class -> ROAD_CLASS
• featureID -> FEATUREID
• numberOfLanes -> ROAD_LANES
• routeNumber -> ROUTE_NUM
• routeType -> ROUTE_TYPE
• type -> ROAD_TYPE
• name -> PARK_NAME
• name_alt -> NAME_ALT
• STOPABBR -> STOP_ABBR
• STOPNAME -> STOP_NAME

Remove:

• id
• ATSTREET
• BAY
• CORNER
• FACILITY_T
• GIS_ID
• ONSTREET
• PLACEMENT

6. Run the Workspace

Below is an image of the final workspace. Run the workspace and check that the output files contain the correct attribute schema.

Debugging Tip

The feature types defined by each selected schema source will be logged. For example:

Reader 'SHAPEFILE_1' of type 'SHAPEFILE' provided schema for types: BusStops,CityParks,Roads

If features are destined for a Feature Type and that Feature Type is not defined in the Schema Sources, the following type of error message will appear near the bottom of the log file:


 

Additional Resources

• Adding Readers as a Resource

Data Attribution

The data used here is adapted from data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.