Workbench Git Integration from the Command Line

FME also supports Github integration in the FME Flow via Version Control. Version Control enables you to save and access previous versions of repositories seamlessly. However, this article does not cover enabling Version Control in FME Flow. Instead, the focus here is on Git integration for FME Form and local workspace development.

Introduction

Git integration with FME Workbench enables seamless support for CI/CD pipelines, streamlining version control and collaboration for more efficient development and reliable deployments. This integration allows you to use Git tools such as TortoiseGit, Sourcetree, or command-line interfaces to work directly with the Compare Workspace tool, view differences (git diff), and resolve conflicts (git merge).

This tutorial provides step-by-step guidance on creating, versioning, comparing, and resolving conflicts in your workspace using Git Bash. A repository with shell scripts, workspaces, and data is available in Safe Software’s public Github space to help you set up and test your Git environment.

Requirements

• FME Form 2023.0+
• Tutorial workspace: Version1.fmw 
• Git 

Step-by-Step Instructions

Part 1: Set Up a Git Environment

These instructions are for use with Git Bash. In a Windows environment, ensure Bash is enabled for your Git installation. 

1. Create Custom diff/merge Shell Scripts

Inside Safe Software's Workspace Comparison Git Configuration repository, there are two shell scripts: workspace_diff.sh and workspace_merge.sh. These scripts are designed to do a diff and merge workspaces, respectively.  
Workspace_diff.sh

#!/bin/bash
# Arguments are: path old-file old-hex old-mode new-file new-hex new-mode

echo "Comparing changes on '" $1 "' using FME Workspace Compare"

fmeworkbench -TITLE-OVERRIDE "[$1] Compare Changes" -COMPARE-TITLE1 "Left" -COMPARE-TITLE2 "Right" -COMPARE $5 $2
exit 0

Workspace_merge.sh

#!/bin/bash
# Arguments are: ancestor-file current-file other-file path
echo "Launching FMEWorkbench to merge " $4 

fmeworkbench -TITLE-OVERRIDE "[$4] Merge" -COMPARE-BASE $1 -COMPARE-BASE-TITLE "Base" -COMPARE-TITLE1 "Ours" -COMPARE-TITLE2 "Theirs" -COMPARE $2 $3

Create or save a copy of the two shell scripts to a folder on your local machine. 

2. Edit PATH Variables
In your system's environmental variables, edit your PATH variable to include paths to both the shell scripts folder and the FME Form installation directory.

Refresh or open a new console window and ensure the PATH variable contains the new paths. 

echo $PATH

3. Edit gitconfig

Open your gitconfig file in a text editor (TIP: In the console, git config --list --show-origin will list the path). 

Register the shell scripts as the custom drivers called “fmeworkbench” by adding the following to your gitconfig file.

[diff "fmeworkbench"]
	command = workspace_diff.sh
	binary = true
[merge "fmeworkbench"]
	name = FME Workbench Merge Driver
	driver = workspace_merge.sh %O %A %B %P

4. Create a Local Workspace Repository 
Create a local repository where you will keep track of versioned workspaces (e.g. "WorkspaceRepo").

In the console, navigate to the repository. Enable change tracking in this repository.

git init

The "WorkspaceRepo" will now contain a ".git" folder. If you need to restart the tutorial, delete this folder and reinitialize. 

5. Add gitattributes

To ensure FME Workbench can parse workspace files, Git must not attempt to merge them as plain text. Instead, the repository will be configured to use FME Workbench Compare tool for comparisons.

Use a text editor to create a new .gitattributes file in your workspace repository (e.g., "WorkspaceRepo").

Inside the text file, add the following commands.

*.fmw diff=fmeworkbench merge=fmeworkbench 

(TIP: When saving, do not add a file extension, as this may cause Git to ignore it)

Part 2: git diff

"git diff" launches the FME Workbench Compare Workspace tool. This is a tool that visually compares .fmw files, highlighting transformers, connections, and parameter changes. You might use git diff to compare two versions of the same workspace at different stages of the development cycle or worked on by separate authors.

1. Create a Workspace

Place the tutorial workspace (Version1.fmw) in the designated git repository.

2. Make an Initial Commit

Stage the first iteration of the Version1.fmw workspace.

git add Version1.fmw

Commit the workspace.

git commit -m "Create Version1.fmw"

The unedited workspace is now commited. 

2. Check Out a New Branch 

Next, we will check out a branch to commit changes to. 

git checkout -b addInspector

3. Make Changes to the Workspace
From the console, launch the Version1.fmw workspace in FME Workbench. 

fmeworkbench Version1.fmw

FME Workbench should automatically open. 

Add an Inspector transformer to the canvas. Connect it to the PostalAddress Reader Feature Type. 


Save the workspace and close FME Workbench. 

Stage and commit the changes.

git add Version1.fmw
git commit -m "Add Inspector to Version1.fmw"

4. Compare Changes (git diff) 

Navigate back to 'main' branch and run a 'git diff' command to compare the original workspace (main) and the updated workspace (addInspector).

git checkout main
git diff addInspector

'git diff' will launch the FME Workbench Compare Tool. The original workspace ('main' branch) is on the left. The edited workspace ('addInspector' branch) is on the right. Updates, like the added Inspector transformer, are highlighted in different colors. For a full overview of the Compare tool, review the Documentation. 

Without saving, exit FME Workbench.

At this stage, we could merge the 'addInspector' updates into the 'main' branch without issue. However, to illustrate the process of resolving conflicts between workspace versions, proceed to Part 2 without performing the merge.

Part 3: git merge

'git merge' launches the FME Workbench Compare tool to resolve conflicts between workspace versions. It is commonly used to combine workspaces developed separately, ensuring all updates are integrated into a single version. Alternatively, you might use it after a regression to identify changes that could have introduced an error. Regardless of the scenario, the merge tool provides a detailed comparison of two workspace versions, enabling precise conflict resolution.

1. Edit the Workspace on a New Branch

Create a new branch and launch a new version of the workspace in FME Workbench.

git checkout -b editSpatialFilter
fmeworkbench Version1.fmw

In FME Workbench, you’ll notice that the Inspector is missing. This is because the workspace was launched from the 'main' branch, which hasn’t yet been merged with the changes from the 'addInspector' branch.

In this new workspace, open the SpatialFilter transformer. In this example, we'll update the parameters. In Tests to Perform, add the rest of the 'OGC' filters. In Pass Criteria, update the value to 'Pass Against All Filters". 

Close the SpatialFilter. Save and close FME Workbench. 

2. Stage, Commit, and Merge Changes. 

Stage and commit the changes.

git add Version1.fmw
git commit -m "Add additional OGC tests to SpatialFilter'

Launch the FME Workbench Compare tool to verify the workspace changes between 'main' and 'editSpatialFilter'. To compare two specific branches, use the format "git diff <branch A> <branch B>".

git diff main editSpatialFilter

The changes to the SpatialFilter transformer will be highlighted. 

Without saving, close FME Workbench.

3. Merge Workspaces Without Conflicts. 

To begin the process of merging the changes from both branches, return to 'main' branch.  

First, merge the addInspector branch. 

git checkout main
git merge addInspector

The addInspector branch should merge without conflicts, as the workspace only has an additional Inspector transformer that does not conflict with anything on the main branch. 

4. Merge Workspaces With Conflicts. 

Next, merge the editSpatialFilter branch. 

git merge editSpatialFilter

Merging the editSpatialFilter branch will result in a conflict due to conflicting differences between the workspaces: the absence of the Inspector transformer and updated SpatialFilter parameters. 

To resolve this conflict, FME Workbench will automatically launch.

The Compare tool in Merge mode offers three different tabs for viewing conflicts between workspaces:  

• "Ours vs. Theirs": Compare your workspace ("ours") with the conflicting version ("theirs"). Use "Copy to Left" to resolve conflicts and merge changes directly in your Git tool.
• "Base vs. Ours": Shows the original workspace ("base") and your version ("ours"). This view is informational; neither version can be edited.
• "Base vs. Theirs": Shows the original workspace ("base") and the conflicting version ("theirs"). This view is also informational and not editable.

From the "Ours vs. Theirs" tab, select Details next to the SpatialFilter. This opens a window that displays the changes. 

After review, close the window. With the SpatialFilter still selected, click "Copy to Left" to merge the changes with the left side ("Ours"). 

FME will apply the changes and refresh the Compare windows. 

Save the workspace. Exit FME Workbench. A dialog window may open asking to confirm the changes to "Ours" as the final merged workspace. Select Yes to continue. This merges the changes made to 'editSpatialFilter' into 'main'.

Git will prompt you to enter a message to justify the merge (TIP: To write a message, press "i". When finished, press "esc". Write ":wq" and press enter)

Git will return a message confirming the merge.

5. Confirm Merges

We can view a history of using the FME Compare Tool with Git. 

git log --oneline --graph

Git returns a visual representation of the commit history, including merges, branches, and the relationships between commits.

Use this process for other FME Workspaces you want to manage with Git. Integrating Git into your workflow makes it easier to track changes, resolve conflicts, and collaborate on projects. As you get more comfortable with these tools, managing version control will feel simpler and more efficient.

Explore other FME tools for CI/CD at Getting Started With CI/CD.