Introduction
A dynamic workflow references schema flexibly from the source, destination, and/or external sources, rather than being locked to predefined, fixed schema definitions. This schema-independent approach allows workflows to adapt by either determining the schema at runtime or, when available, incorporating fixed schema definitions (such as a reader resource). This allows dynamic workflows to handle varying inputs, reducing long-term maintenance requirements.
Files
- DynamicWorkflow.zip (28.4 MB)
Dynamic Workflow Basics
Basics of Dynamics: See this article for more information about dynamic workflows, including common uses and components.
Tutorial Series
The following tutorials display different methods for creating dynamic workflows. When creating a dynamic workflow, the destination Feature Types must know how to obtain the desired output schema. The source of this schema differs across the examples. The workflow is used when the desired schema differs, as shown below.
|
Dynamic Workflows: Destination Schema as a Mirror Image of the Source Dataset: Demonstrates a simple dynamic workspace. This type of workflow can output an exact copy of the Source Feature Type schema when the schema is not known in advance. |
|
| Dynamic Workflows: Destination Schema is Derived from an External Dataset: Demonstrates using workspace resources with an external dataset to provide a destination schema. This type of workflow is used when the desired schema is different from the source schema. | |
| Dynamic Workflows: Destination Schema Contains Different Feature Type Names: Demonstrates how to handle a Source Schema and Destination dataset with different feature type names. When this occurs we need to use a special parameter called the 'Schema Definition Name'. The example also uses a database as the external Dataset. | |
|
Dynamic Workflows: Destination Schema is Derived from a Lookup Table: Demonstrates using an external lookup table to provide a destination schema. This type of workflow is useful when maintaining a schema outside of the FME environment. |
|
| Dynamic Workflows: Destination Schema is Derived from a Schema Feature: Demonstrates using list attributes in a workspace to provide a destination schema. This is useful when defining the destination schema dynamically within the workspace. | |
| Dynamic Workflows: Destination Schema Derived from Multiple Sources: Demonstrates using List Attributes in a workspace to provide a destination schema when there are multiple source schemas. This is useful when combining multiple source datasets with unknown schemas or adding unknown attributes to a static schema. |
Additional Articles
Advanced Technique - Using the SchemaMapper
Discusses the use of the SchemaMapper transformer in a dynamic setting. This transformer can simplify the workflow and consolidate all schema mapping details into an external source.
Advanced Technique - Modifying the Schema Feature
Using the SchemaSetter custom transformer to build a schema feature, the feature will be manipulated using FME. This is a common pattern and should be used whenever you need to build a schema and edit its contents.
Dynamic Workflows: Dealing with Schema Drift
Using the SchemaScanner and ChangeDetector transformers, we can detect schema changes and alert the user before any changes are made to existing dataset schemas.