Files
Introduction
This article is part 2 of a 5-part series to help you get started with basic Python and FME. This article discusses the use of Startup Python Script functionality within Workbench and provides two examples. The first uses a startup Python script to copy a file before running a translation, and the second writes custom (user-defined) messages to the log file.
Adding a Python startup script to your workspace is a great way to extend the functionality of Workbench. You may want to set up the environment in a certain way before a translation is run, or you may want the increased control over FME that a custom script offers. You can define scripts directly within Workbench. In the Workbench Navigator pane under Workspace Parameters > Scripting, you will see a parameter called Startup Python Script. Right-click on the parameter and select Edit Parameter Value to open a special Python editing window.
Here, you can enter some Python code that will execute after the log file has been opened and the mapping file has been completely parsed, including the execution of scripted parameters, but before the readers have begun to do their processing. It is also possible to define global Python variables in a startup Python script that can be accessed later on in the translation within a Shutdown Python Script.
This is not the place to set or return an FME parameter—see the Scripted Parameters article for more details.
Examples
Example 1: Create a .zip File Backup of a Source Dataset
Attached workspace: StartupPython1.fmwt in startup-workspaces-2021.zip
You may want to create backups of your data before or after you run a translation. It is possible to do so with a startup Python script or a shutdown Python script. In this example, you will be creating a zip file backup of your shapefile input dataset. You can use the attached workspace template, StartupPython1.fmwt in startup-workspaces-2021.zip (see the Files section for the link).
Open the attached workspace. The workspace reads the input shapefile dataset and reprojects the data before writing the data to GML. Before we run the workspace, we want to create a zip file from the source dataset and place it in a TEMP folder on the C: drive if it does not exist already. If a zip file with the same name already exists, the file will not be copied, and a message will be printed to the log window. Go to the Navigator > Workspace Parameters > Scripting > Startup Python Script to view the script or see the following.
# import fme and shutil modules
import fme, shutil
from os import path
# Filepath hardcoded to C:\TEMP
workspacePath = 'C:\\TEMP'
# Filepath of source dataset using macroValues dictionary
sourceFile = fme.macroValues['SourceDataset_SHAPEFILE']
# Gets dataset name without extension with '_backup' appended
sourceName = f'{path.splitext(path.basename(sourceFile))[0]}_backup'
# Path of new zip with name
zipDir = f'{workspacePath}\\{sourceName}'
# Root directory of folder containing source dataset
sourcePath = path.dirname(sourceFile)
# If zip file with same name already exists, do not create, add message to log window
if path.exists(f'{zipDir}.zip'):# Message to log window
print(f'{zipDir}.zip already exists. Did not create zip.')
else:
# Use shutil module to create zip file
shutil.make_archive(zipDir, 'zip', sourcePath)
print(f'Created backup of {path.basename(sourceFile)}')
print(f'Path at {zipDir}.zip')
# Debugging
# print(sourceName)
# print(zipDir)
The script imports the fme, shutil modules and the path class from the os module. The shutil.make_archive() (line 23) requires at least 3 arguments:
- The name of the file to create, including the path
- The format of the archive file (zip in this example)
- The folder containing the data to zip
We can use Python functions and FME macros to find all the first and third arguments. In this example, we want to create a zip file of the source datasets in the C:\TEMP folder with the same name as the shapefile and the text ‘_backup’.
To find the shapefile name, we can use the fme.macroValues[] dictionary to find the file path of the source dataset using the source dataset parameter (sourceFile in script) and get only the last part of the file path.
In the example, it is ‘SourceDataset_SHAPEFILE’. Your source dataset parameter may have another name. You can find the name by right-clicking on User Parameters in the Navigator pane and selecting Manager User Parameters, where you can see the Parameter Identifier.
To get the dataset name without the extension (sourceName in script), we can use the path.basename() function from the os module to find the file name. The path.splitext() function will split the text at the first ‘.’ and as we are only interested in the file name and not the extension, we only return the first piece of text with [0]. We append ‘_backup’ to the end of the file name.
We finish creating the full zip path by taking the destination folder (hard-coded as workspacePath), adding ‘\\’, the file name (sourceName), and adding ‘_backup’. Note that because the backslash character is an escape character in Python, we need to use double backslashes in file paths.
Next, we need to find the folder containing the data to zip because a shapefile is a folder-based format, so you cannot simply copy only the .shp file. As we have already found the file path of the source dataset file, we can use the path.dirname() function to get the file path of the folder containing the data (sourcePath in script).
To prevent overwriting existing datasets with the same name, the zip file backup of the source dataset is only created if it doesn’t already exist. If it does, only a message will print to the log. This is controlled by an if/else statement. The last three lines are print functions for debugging purposes.
Run the workspace (Run > Run Workspace or press F5). After the workspace has finished running, go to the C:\TEMP folder. You should see a zip file with your input dataset name with ‘_backup’ at the end.
You do not have to write to C:\TEMP. You can change the script to write to another hard-coded file path instead or use a Published parameter to ask a user to select where they want the zip file to be written to every time they run the workspace.
Example 2: Custom Log Messages with Startup Python Script
Attached workspace: StartupPython2.fmw in startup-workspaces-2021.zip
A log file contains a lot of information already, but what if you want to write custom messages to the log file? You can add custom messages to the log file with a startup Python script, a PythonCaller transformer, and a shutdown Python script. In this example, you will add a short message to the log file with a startup Python script.
We can write custom messages to the log file before running FME with the following startup Python script (please see StartupPython2.fmw in startup-workspaces-2021.zip):
# Import FME Objects module
import fmeobjects
# Create FME Log File object name called logger
logger = fmeobjects.FMELogFile()
# Call logMessageString method to add custom text and set WARN severity level
logger.logMessageString("You can add custom text with custom severity level to the log file",1)
# Demonstrates print adds message with INFORM severity level
# In FME versions older than 2018.1, print function will only print to log pane
print('This will only print to the Translation Log pane')
You can access the log file with the FME Objects module, so we need to use import fmeobjects. A log file object named ‘logger’ is created by calling the fmeobjects.FMELogFile() class. The object then calls the logMessageString() method to add custom text with a specified severity level to the log file.
Beginning in FME 2018.1+, the print function will output the text to both the log file at the INFORM severity level and the translation log pane.
Open the attached workspace. It should look very similar to the first example. Run the workspace (Run > Run Workspace or press F5). Go to the Translation Log pane and right-click within the pane. Select ‘Find’ from the menu and type ‘log’. You should see instances of the text ‘log’ highlighted within the Translation Log pane. You will see the custom log message and the print message highlighted in the pane.
Documentation
To access Help documentation for Startup and Shutdown scripts within Workbench, please go to: Startup and Shutdown Python Scripts
For a complete list of FME variables available for Startup Python scripts, please go to: FME_BEGIN_PYTHON
You can find complete documentation of the Python FME Objects API here: FME Objects Python API
Data Attribution
The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.