Tutorial: Dynamic Workflows

Introduction

A dynamic workflow is when the source and destination feature types in a workspace are not locked to a specific schema. A schema-independent workspace can be beneficial because it can handle varying inputs and requires minimal long-term maintenance. It can read in multiple feature types from a source dataset, even when they are structured differently. The destination schema will be automatically determined at runtime based on the source schema or a user-defined template (like a dataset, database, or lookup table).

Dynamic Workflow Basics

Basics of Dynamics: See this article for more information about dynamic workflows, including common uses and components.
 

Tutorial Series

The following tutorials display different methods for creating dynamic workflows. When creating a dynamic workflow, the destination Feature Types must know how to obtain the desired output schema. The source of this schema differs in each example below.

Dynamic Workflows: Destination Schema as a Mirror Image of the Source Dataset: Demonstrates a simple dynamic workspace. This type of workflow can output an exact copy of the Source Feature Type schema when the schema is not known in advance.
Dynamic Workflows: Destination Schema is Derived from an External Dataset: Demonstrates using workspace resources with an external dataset to provide a destination schema. This type of workflow is used when the desired schema is different from the source schema.
Dynamic Workflows: Destination Schema Contains Different Feature Type Names: Demonstrates how to handle a Source Schema and Destination dataset with different feature type names. When this occurs we need to use a special parameter called the 'Schema Definition Name'. The example also uses a database as the external Dataset.
Dynamic Workflows: Destination Schema is Derived from a Lookup Table: Demonstrates using an external lookup table to provide a destination schema. This type of workflow is useful when maintaining a schema outside of the FME environment.
Dynamic Workflows: Destination Schema is Derived from a Schema Feature: Demonstrates using list attributes in a workspace to provide a destination schema. This is useful when defining the destination schema dynamically within the workspace.
Dynamic Workflows: Destination Schema Derived from Multiple Sources: Demonstrates using List Attributes in a workspace to provide a destination schema when there are multiple source schemas. This is useful when needing to combine multiple source datasets of an unknown schema or add unknown attributes to a static schema.

Additional Articles

Advanced Technique - Using the SchemaMapper
Discusses the use of the SchemaMapper transformer in a dynamic setting. This transformer can be used to simplify the workflow and move all the schema mapping details into an external source.

Advanced Technique - Modifying the Schema Feature
Using the SchemaSetter custom transformer to build a schema feature, then the feature will be manipulated using FME. This is a common pattern and should be used whenever you need to build a schema and edit its contents.

Dynamic Workflows: Dealing with Schema Drift
Using the SchemaScanner transformer and the ChangeDetector transformer, we can detect changes in the schema and alert the user before any changes are made to existing dataset schemas.
 

Additional Resources

• Definition of Schema
• Dynamic Workspaces Help Documentation
• Feature Type Fanouts