Test Batch Processing Method 3: Using the WorkspaceRunner

Tandra Geraedts
Tandra Geraedts
  • Updated

FME Version

Introduction

The WorkspaceRunner transformer runs a separate .fmw file in the workspace where it is used. In conjunction with the Directory and File Pathname reader, it can be used for test batch processing.

The same technique can be carried out on FME Flow by using the FMEFlowJobSubmitter transformer instead of the WorkspaceRunner. Furthermore, FME Flow Automations is a powerful and versatile tool for enterprise batch processing. For more information on test batch processing or to learn about enterprise batch processing, see FME Batch Processing Methods.

 

Step-by-Step Instructions

In this example, the user has a polygon-building workspace that needs to be used on a number of MIF/MID datasets stored within a single folder. To accomplish this, a new workspace using the WorkspaceRunner transformer can be created.

 

Workspace 1: PolygonBuilder.fmw

1. Add a MapInfo MIF/MID Reader

Add a MapInfo MIF/MID reader to a blank workspace. For the dataset, select any file from within the folder you are trying to batch process. For this example, add the dataset parcel_k24.mif. Before adding the reader, ensure you select Single Merged Feature Type under the Workflow Options. This will ensure all of the datasets within the folder containing parcel_k24.mif will get processed.

AddReader.png

 

After adding the reader to the canvas, go back into the parameters, and in the Format Attributes tab, enable fme_basename. 

ExposeFMEBasename.png

 

2. (Optional) Add a GeometryFilter Transformer

The MapInfo MIF files can contain multiple geometry types. For this example, we are only interested in the lines. Add a GeometryFilter transformer to the canvas, and in the parameters, set the Geometry Types to Filter to Line.

 

3. (Optional) Create Polygons

Now that we have filtered for lines, we need to connect the lines together to create polygons. Add an AreaBuilder transformer to the canvas, and in the parameters, enable Group Processing, then select fme_basename to Group By. The rest of the parameters can be left as the default.

Areabuilder.png

 

4. Write to Shapefile

Add a Shapefile writer to the canvas, and set the Dataset to your output folder location. Set the Shapefile Definition to Automatic. When the Feature Type dialog appears, just click OK. Connect the writer feature type to the AreaBuilder. Once connected, open the writer feature type parameters again, and change the Shapefile Name to fme_basename and the Geometry to shape_polygon.

FeatureType.png

 

5. Save and Close the Workspace

You can test-run this workspace to ensure it works if you like. We will be using another workspace to run this workspace, so for now, you can just save it to a location you can find, close it, and then continue with Workspace 2.

Canvas.png

 

Workspace 2: WorkspaceRunner.fmw

1. Add a Directory and File Pathnames reader

In a blank workspace, add a Directory and File Pathnames reader and set the dataset to the Directory (folder) containing all the MapInfo MIF files you wish to batch process. For this example, it will be the Data folder. Be sure to switch the Workflow Options back to Individual Feature Types. Click OK to add the reader. 

AddReaderWorkspaceRunner.png

 

2. Add a WorkspaceRunner Transformer

Add a WorkspaceRunner transformer to the canvas and connect it to the PATH reader feature type. The WorkspaceRunner will run another workspace when the workspace containing it runs. In the parameters, set the FME Workspace to the workspace you created in Workspace 1, for this example it will be PolygonBuilder.fmw. Once you select a workspace, the parameters at the bottom will populate. We will need to change these to ensure they work correctly. Change the Source MapInfo MIF/MID File(s) to the attribute path_windows. Then, change the Destination Esri Shapefile Folder to your output folder.

WorkspaceRuner.png

 

3. Connect Loggers

To log any errors that might arise with test batch processing, right-click on the WorkspaceRunner and click Connect Loggers. Since the WorkspaceRunner has three output ports, three Loggers will be connected. 

WorkspaceRunnerCanvas.png

 

Run the workspace and then confirm in your output folder that the files were processed.

log.png

 

As the WorkspaceRunner runs, a number of FME processes are started - one for each file. The WorkspaceRunner can be set to wait for each process to complete before starting the next, in which case there would only be one new process at a time. There is a parameter in the WorkspaceRunner for setting the Maximum Number of Concurrent Processes. This limits the number of concurrently executing workspaces to the value specified. If specified, this parameter value must be an integer in the range 1-7. The specified value includes the process executing WorkspaceRunner. For example, if the value is set to 7 then WorkspaceRunner can only start 6 additional FME processes concurrently. Checking the display in the Task Manager will show the number of executing fme.exe processes.

TaskManager.png

Shapefiles.png

 

Data Attribution

Data provided by the Travis County Open Data Portal and GeoHub

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.