Batch Processing Method 1: Command Line or Batch File

FME on the Command Line

FME can be run from the command line in Windows: Opening the Command Prompt window and typing FME gives the full list of options displayed below.

The command FME <workspacename>.fmw will run a particular workspace. Therefore, if FME workspaces can be called from the command line, a series of workspaces (or the same workspace with differing datasets) can be called from a DOS batch (.bat) file.

Usage

Usage: fme <controlFile> [<keyword> <value>]* [--<macroName> <value>]*
fme <scriptfile> [<scriptArgument>*]
fme <licenseFile>.fmelic
fme <command> <arguments>
where:
<controlFile> is one of <mappingFile>.fme or <workspace>.fmw
<scriptFile> is one of <tclScript>.tcl or <pythonScript>.py
<command> is one of:

Command Name Arguments
------------ -----------------------------------------
GENERATE <sourceType> <destType> <sourceDataset> <controlFile> [<keyword> <value>]*
PARAMETER_FILE <parameterFile>
COMMAND_FILE <commandFile>
REGISTER_SOCKET <hostName> <service> [serverConfigFile] [-<ServerParmName> <ServerParmValue>]*
CREATE_SOCKET <service> [serverConfigFile] [-<ServerParmName> <ServerParmValue>]*
GENTRANS [<keyword> <value>]* <parameterFile>
GENTRANS [<keyword> <value>]* <sourceType> <sourceDataset> <destType> <destDataset>
LIST_TRANSFORMERS [VERBOSE]
LIST_UNLICENSED_TRANSFORMERS
LIST_FACTORIES
LIST_FUNCTIONS
PROTECT <sourceFile> <destFile>

Q) How do I pass the source and destination datasets to the batch process?

A) If you are processing an FME workspace then you don't necessarily need to define the source and destination in your batch command - they will already be defined in the workspace.

However, for documentation purposes and for workspaces where the dataset hasn't been defined, the source and destination datasets are held as parameters (or Macros) in FME. A parameter is a type of variable which is passed to the workspace at runtime. By default the source and destination datasets are set as parameters and you can define the values for the parameters SourceDataset and DestinationDataset as follows

When you have multiple source datasets, for example, you would need to define SourceDataset_<formatname>_1 where <formatname> is the abbreviated name of the source data format, for example IGDS.

You can pass values to parameters by running an FME workspace from the command line with the following format…

 fme <workspacename>.fmw --<myparametername> <parametervalue>

For example, to pass the value "myfile.dgn" to the parameter "SourceDataset_IGDS_1" in workspace project1

 fme project1.fmw --SourceDataset_IGDS_1 myfile.dgn

Q) How can I pass multiple source and destination datasets to the batch process?

A) Using a bat file with DOS variables. For example...

 for %%f in (*.dgn) do "c:\program files\fme\fme.exe" C:\FMEInput\dgn2dxf.fme --SourceDataset "%%f" --DestDataset "%%~nf.dxf"

Q) My command line doesn’t work. It is very long and I think the Operating System doesn’t like that! Is there any way around long command lines?

A) Yes. The command line parameters can be stored in a file and read by FME using the syntax…

 fme PARAMETER_FILE <parameterFile>

The parameter file must contain all the values required to run FME. If you create a parameter file called 'myparamfile.par'. The contents of the parameter file will look something like:

myworkspace.fmw --PARAM1 value1 --PARAM2 value2

and run as:

fme PARAMETER_FILE myparamfile.par

Unfortunately, you can't mix PARAMETER_FILE and other command line parameters, so:

fme PARAMETER_FILE myparamfile.par --PARAM3 value3

is invalid and FME will ignore PARAM3

FME and DOS Batch Files

The attached example (See Attachment: translate.bat in CompleteBatchExample.zip) translates a directory of DWG files to GML files. A validation translation is run first, then the files that pass validation are translated to GML. Any file that fails validation or translation (this will also deal with fme.exe crashing) are placed in a FAILED directory. All other files are placed in the COMPLETED directory. A separate log file for each translation is placed in the LOGS directory.

A report file is also generated giving the results (passed or failed) of each translation, and indicates the reason for a failed translation.

If a translation fails, the report file is emailed to a user.

The moveFilesBack.bat file places the DWG files back into the SOURCE directory, so the translation can be tested again.

The following commands, characters, and variables were used in this example.

Useful DOS Commands

DIR

Lists the files within a directory. DIR /B can be used to create a list of files.

FIND (FINDSTR)

Used to search for specific phrases within a log file. The FIND command has been replaced by the FINDSTR command in newer releases of Windows.

FOR

Used for running a translation using multiple files. Each file uses its own translation.

IF

Performs conditional processing in batch programs.

ECHO

Used to send text to the screen or a file. ECHO. gives a blank line.

MOVE

Moves files from one directory to another.

MD

Used to create directories. FME sometimes runs into problems if an output directory does not already exist.

DEL

Used to delete files that are no longer necessary. DEL /Q %LOG%\*.* will delete all log files, without asking permission.

PAUSE

Useful during debugging. Used to prevent the DOS window from closing once the translation is done.

SET

Used to set variables. Useful for setting directory names so that there is a single place to edit them, and avoids needing to type them in more than once.

SetLocal EnableDelayedExpansion/EndLocal

If !ERRORLEVEL! of FIND is to be used SetLocal EnableDelayedExpansion must be at the beginning of the file, and EndLocal at the end. This delays the evaluation of ERRORLEVEL, otherwise it is done when the batch file starts, and ERRORLEVEL is not really the ERRORLEVEL of FIND.

Useful DOS Characters

^

This is an escape character. It can escape carriage-returns, so it can act as a continuation character (just make sure you don't place a space after it).

 fme acad2gml_load.fmw --SourceDataset_ACAD "%%F"^                       --DestDataset_GML "%OUTPUT%\%%~nF.gml"^                       --LOG_FILE "%LOG%\%%~nF_load.log"

is the same as

 fme acad2gml_load.fmw --SourceDataset_ACAD "%%F" --DestDataset_GML "%OUTPUT%\%%~nF.gml" --LOG_FILE "%LOG%\%%~nF_load.log"

|

Used to pipe the results of one command into another command. FME | FIND "Build" will find the build number of FME.

>

Sends the output of a command (such as ECHO) to a new text file. This will overwrite an existing file.

>>

Appends the output of a command (such as ECHO) to a new text file. This will NOT overwrite an existing file.

::

Used to create a comment line. This is superior to using REM--it is faster and it does not have limitations to what characters are used in the comment.

@

Prevents a command from being echoed, even if ECHO is set to ON.

Useful Variables

!ERRORLEVEL!

Used to return the ERRORLEVEL of FIND. A value of 0 means that the search string was found, and a value of 1 means that it wasn't. Use this instead of %ERRORLEVEL%. This variable is expanded when the command is run, not when the batch file starts.

%DATE%

Returns the current date. Useful for creating timestamps.

%TIME%

Returns the current time. Useful for creating timestamps.

%CD%

Returns the current directory. The directory the batch file is being run from.

Password

I usually create a PW environment variable (user variable) which contains my password. Then I can call my password in the batch file using %PW%, instead of adding my password to the actual batch file.