FME Batch Processing Methods

Introduction

Currently, FME has two options for batch processing - test and enterprise. In FME Form, you can build your workspaces and test batch processing, and when you're ready, you can deploy to FME Flow for enterprise batch processing. 

Enterprise Batch Processing

FME can run workspaces on command, whether manually triggered, on a schedule, or based on a file or directory update. These batch-processing methods are built into FME Flow Automations. Automations are set up on a canvas similar to Form’s workbench: there is a drag-and-drop canvas, and it involves no coding. FME Flow takes the authored workspaces and enables users to run workspaces with event-triggered workflows like schedules and directory updates.

Another feature of enterprise batch processing in FME Flow is dynamic workspaces. This feature allows information to be passed from one workspace to another through the automations writer. Workspaces can be authored as generic templates and run for varying inputs. Parallel Processing and Workspace Chaining are two examples of this functionality.

Lastly, FME Flow allows users to perform complex job orchestration to enable users to get the most out of their environment. These batch processes can be triggered and adjusted to external inputs, resulting in logging, notification, and flexible retry logic so FME doesn’t miss a beat.  

For more information on enterprise batch processing methods, please see the following tutorials:

• Job Orchestration with Automations
• Dynamic Workspace Chaining in FME Flow Automations
• Dynamic Workspaces: Data Driven Parallel Processing in FME Flow Automations

Test Batch Processing

Users can perform Test Batch Processing in FME Form, and when they are ready to deploy to FME Flow, they can use the Enterprise Batch Processing methods. Methods for test batch processing include using the Command Line, Command File, or WorkspaceRunner transformers. 

For more information on test batch processing methods, please see the following tutorials:

• Command Line or Batch File
• Command File
• WorkspaceRunner Transformers

WorkspaceRunner transformers are incompatible with workspaces running on FME Flow. The FMEFlowJobSubmitter is required in place of the WorkspaceRunner. However, to get the most versatility from FME Flow, the FME Flow Automations Writer allows you to pass parameters from one workspace to another, which is the currently recommended method.

Additional Resources

Self-paced training course: FME Flow Authoring

Enterprise Integration Patterns