Batch Processing Using the WorkspaceRunner

Introduction

The WorkspaceRunner transformer runs a different workspace from within another workspace. In conjunction with the Directory and File Pathname reader, it can be used as a device for batch processing.

The same technique can be carried out on an FME Server by using the FMEServerJobSubmitter transformer in place of the WorkspaceRunner. When publishing a workspace to FME Server that includes a WorkspaceRunner transformer is not recommended. The transformer will try to start an FME instance outside of FME Server to run the workspace, and this FME instance will require an additional license.

In this example, the user has a polygon building workspace that needs to be used on a number of MIF/MID datasets stored within a single folder. This can be accomplished by creating a new workspace that uses the WorkspaceRunner transformer.

Step-by-step Instructions

Workspace 1: PolygonBuilder.fmw

1. Add a MapInfo MIF/MID Reader

Add a MapInfo MIF/MID Reader to a blank workspace, for the dataset, select any file from within the folder you are trying to batch process. For this example, add the dataset parcel_k24.mif. Before adding the reader ensure you select Single Merged Feature Type under the Workflow Options, this will ensure all of the datasets within the folder containing parcel_k24.mif will get processed. After adding the reader to the canvas, go back into the parameters, and in the Format Attributes tab, enable fme_basename.

MapInfo MIF/MID reader, set the Workflow Options to Single Merged Feature Type

MapInfo MIF/MID reader parameters, under the Format Attributes tab, enable fme_basename

2. (Optional) Add a GeometryFilter transformer

The MapInfo MIF files can contain multiple geometry types, for this example, we are only interested in the lines. Add a GeometryFilter transformer to the canvas, and in the parameters set the Geometry Types to Filter to Line.

3. (Optional) Create polygons

Now that we have filtered for lines, we need to connect the lines together to create polygons. Add an AreaBuilder transformer to the canvas, and in the parameters select fme_basename to Group By, the rest of the parameters can be left as the default.

Create polygons by Grouping By fme_basename

4. Write out to Shapefile

Add a Shapefile writer to the canvas, and set the Dataset to your output folder location. Set the Shapefile Definition to Automatic. When the Feature Type dialog appears, just click ok. Connect Writer to the AreaBuilder. Once connected open up the parameters again, this time change the Shapefile Name to fme_basename and the Geometry to Shape_Polygon.

To fanout datasets, set the Shapefile Name to fme_basename

5. Save and close workspace

You can test run this workspace to ensure it works if you like. We will be using another workspace to run this workspace, so, for now, you can just save it to a location you can find, and then close it, then continue with Workspace 2.

PolygonBuilder.fmw workspace, this is the workspace that will be run using the WorkspaceRunner

Workspace 2: WorkspaceRunner.fmw

1. Add a Directory and File Pathnames reader

In a blank workspace, add a Directory and File Pathnames reader, and set the dataset to the Directory (folder) containing all the MapInfo MIF files you wish to batch process, for this example it will be the Data folder.

Using the Directory and File Pathnames reader, read the Data folder containing all the MIF files

2. Add a WorkspaceRunner transformer

Add a WorkspaceRunner transformer to the canvas. The WorkspaceRunner will run another workspace when the workspace containing it, is run. In the parameters, set the FME Workspace to the workspace you created in Workspace 1, for this example it will be PolygonBuilder.fmw. Once you select a workspace, the parameters at the bottom will populate, we will need to change these to ensure they work correctly. Change the Source MapInfo MIF/MID File(s) to the attribute path_windows (note: ensure the WorkspaceRunner is connected to the Directory and File Pathnames Reader to see this parameter). Then change the Destination Esri Shapefile Folder to your output folder.

Set the FME Workspace to PolygonBuilder.fmw, change the Source MIF File to path_windows

3. Connect Loggers

To log any errors that might arise with batch processing, right click on the WorkspaceRunner and click Connect Loggers. Run the workspace and then confirm in your output folder that the files were processed.

WorkspaceRunner.fmw, with loggers connected to the WorkspaceRunner

Translation Log from the WorkspaceRunner.fmw showing one of the MIF files being translated in the PolygonBuilder.fmw workspace.

As the WorkspaceRunner does its stuff, a number of FME processes are started - one for each file. The WorkspaceRunner can be set to wait for each process to complete before starting the next, in which case there would only be one new process at a time. There is a parameter in the WorkspaceRunner for setting the Maximum Number of Concurrent Processes. This limits the number of concurrently executing workspaces to the value specified. If specified, this parameter value must be an integer in the range 1-7. The specified value includes the process executing WorkspaceRunner. For example, if the value is set to 7 then WorkspaceRunner can only start 6 additional FME processes concurrently. Checking the display in the Task Manager will show the number of executing fme.exe processes.

Windows Task Manager showing three instances of the fme.exe running while running the WorkspaceRunner.fmw workspace.

Sample of the Output folder with the translated Shapefiles

Data Attribution

Data provided by the City of Austin and Travis County, Texas