Startup Python Scripts in FME

Introduction

This article is part 2 of a 5 part series to help you get started with basic Python and FME. This article is about using the Startup Python Script functionality within Workbench and includes two examples. The first uses a startup Python script to copy a file before running a translation and the second writes custom (user-defined) messages to the log file.

Adding a Python startup script to your workspace is a great way to extend the functionality of Workbench. You may want to set-up the environment in a certain way before a translation is run or you may want the increased control over FME that a custom script offers. You can define scripts directly within Workbench. In the Workbench Navigator pane under Workspace Parameters > Scripting, you will see a parameter called Startup Python Script. Right-click on the parameter and select Edit Parameter Value to open a special Python editing window.  Here, you can enter some Python code that will execute after the log file has been opened and the mapping file has been completely parsed, including the execution of scripted parameters, but before the readers have begun to do their processing. It is also possible to define global Python variables in a startup Python script that can be accessed later on in the translation within a shutdown Python script.

Users can enter Python code which will execute before the translation in the Startup Python Script parameter within the Navigator pane under Workspace Parameters > Scripting.

Note: If you want to set or return an FME parameter this is not the place--see the Scripted Parameters article.

Examples

Example 1: Create .zip File Backup of Source Dataset

Attached workspace: StartupPython1.fmwt

You may want to create backups of your data before or after you run a translation. It is possible to do so with both a startup Python script or shutdown Python script. In this example, you will be creating a zip file backup of your shapefile input dataset. You can use the attached workspace template, StartupPython1.fmwt (see Downloads section above for the link).

Open the attached workspace. The workspace reads the input shapefile dataset and reprojects the data before writing the data to GML. Before we run the workspace, we want to create a zip file from the source dataset and place it in a TEMP folder on the C: drive if it does not exist already. If a zip file with the same name already exists, the file will not be copied and a message will be printed to the log window. Go to the Navigator > Workspace Parameters > Scripting > Startup Python Script to view the script or see the following.

# import fme and shutil modules
import fme, shutil
from os import path

# Filepath hardcoded to C:\TEMP
workspacePath = 'C:\\TEMP'
# Filepath of source dataset using macroValues dictionary
sourceFile = fme.macroValues['SourceDataset_SHAPEFILE']

# Gets dataset name without extension with '_backup' appended
sourceName = f'{path.splitext(path.basename(sourceFile))[0]}_backup'

# Path of new zip with name
zipDir = f'{workspacePath}\\{sourceName}'

# Root directory of folder containing source dataset
sourcePath = path.dirname(sourceFile)

# If zip file with same name already exists, do not create, add message to log window
if path.exists(f'{zipDir}.zip'):# Message to log window
    print(f'{zipDir}.zip already exists. Did not create zip.')
else:
    # Use shutil module to create zip file
    shutil.make_archive(zipDir, 'zip', sourcePath)
    print(f'Created backup of {path.basename(sourceFile)}')
    print(f'Path at {zipDir}.zip')

# Debugging
# print(sourceName)
# print(zipDir)

The script imports the fme, shutil modules and the path class from the os module. The shutil.make_archive() (line 23) requires at least 3 arguments:

1. The name of file to create, including the path
2. The format of the archive file (zip in this example)
3. The folder containing the data to zip

We can use Python functions and FME macros to find all the first and third arguments. In this example, we want to create a zip file of the source datasets in the C:\TEMP folder with the same name as the shapefile and the text ‘_backup’.

To find the shapefile name, we can use the fme.macroValues[] dictionary to find the file path of the source dataset using the source dataset parameter (sourceFile in script) and get only the last part of the file path.

In the example, it is ‘SourceDataset_SHAPEFILE’. Your source dataset parameter may have another name. You can find the name by right-clicking on User Parameters in the Navigator pane and selecting Manager User Parameters, where you can see the Parameter Identifier.

The name of the Source Dataset parameter can usually be found by right-clicking the reader in the Navigator pane and selecting Edit User Parameter Definition.

To get the dataset name without the extension (sourceName in script), we can use the path.basename() function from the os module to find the file name. The path.splitext() function will split the text at the first ‘.’ and as we are only interested in the file name and not the extension, we only return the first piece of text with [0]. We append ‘_backup’ at the end of the file name.

We finish creating the full zip path by taking the destination folder (hard coded as workspacePath), adding ‘\\’, the file name (sourceName), and adding ‘_backup’. Note that because the backslash character is an escape character in Python, we need to use double backslashes in file paths.

Next, we need to find the folder containing the data to zip because shapefile is folder-based format so you cannot simply copy only the .shp file. As we have already found the file path of the source dataset file, we can use path.dirname() function to get the file path of the folder containing the data (sourcePath in script).

To prevent overwriting existing datasets with the same name, the zip file backup of the source dataset is only created if it doesn’t already exist. If it does, only a message will print to the log. This is controlled by an if/else statement. The last three lines are print functions for debugging purposes.

The message that prints in the Translation log pane if a file with the same name already exists in the backup location. In this case, the zip file will not be created.

Run the workspace (Run > Run Workspace or press F5). After the workspace has finished running, go to the C:\TEMP folder. You should see a zip file with your input dataset name with ‘_backup’ at the end.

Note: You do not have to write to C:\TEMP. You can change the script to write to another hard-coded file path instead or use a Published parameter to ask a user to select where they want the zip file to be written to every time they run the workspace.

Example 2: Custom Log Messages with Startup Python Script

Attached workspace: StartupPython2.fmw

A log file contains a lot of information already but what if you want to write custom messages to the log file? You can add custom messages to the log file with a startup Python script, PythonCaller transformer, and also a shutdown Python script. In this example, you will be adding a short message to the log file with a Startup Python script.

We can write custom messages to the log file before running FME with the following startup Python script (please see StartupPython2.fmw):

# Import FME Objects module
import fmeobjects

# Create FME Log File object name called logger
logger = fmeobjects.FMELogFile()

# Call logMessageString method to add custom text and set WARN severity level
logger.logMessageString("You can add custom text with custom severity level to the log file",1)

# Demonstrates print adds message with INFORM severity level
# In FME versions older than 2018.1, print function will only print to log pane
print('This will only print to the Translation Log pane')

You can access the log file with the FME Objects module so we need to import fmeobjects. A log file object named ‘logger’ is created by calling fmeobjects.FMELogFile() class . The object then calls the logMessageString() method to add custom text with a specified severity level to the log file.

In FME 2018.1+, the print function will output the text to both the log file at the INFORM severity level and the translation log pane. The print function in FME 2018.0 and older will print messages to the Translation Log pane only--the message does not appear in the translation log file.

Open the attached workspace. It should look very similar to the first example. Run the workspace (Run > Run Workspace or press F5). Go to the Translation Log pane and right-click within the pane. Select ‘Find’ from the menu and type ‘log’. You should see instances of the text ‘log’ highlighted within the Translation Log pane. You will see the custom log message and the print message highlighted in the pane.

The first highlighted line is created with the logger object's logMessageString method. The second highlighted line is created using the print function.

• Go the folder location of where the attached workspace is saved. The log file should be saved in the same folder location (by default) and should have the same name as the workspace but with a .log extension. Open the log file with any text editor. Use the search function to find the custom message within the log. Notice that the print message is not saved in the translation log file.

  

  Notice the text added using the print function does not appear in the log file. Only text added using the logger object's logMessageString method will appear in log files.

  The method of writing custom messages to the log file varies depending on whether it is during the startup or shutdown phase. In a startup script (because FME already has the log file open, and it is dangerous to open it more than once), you should use fmeobjects.FMELogFile().

Documentation

To access Help documentation for Startup and Shutdown scripts within Workbench, please go to: Startup and Shutdown Python Scripts

For a complete list of FME variables available for Startup Python scripts please go to: FME_BEGIN_PYTHON

You can find complete documentation of the Python FME Objects API here: FME Objects Python API

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.