Automating FME Flow Project Synchronization using Jenkins

This article explains how to automate the synchronization of FME Flow projects between two environments (e.g., Dev → Prod) using a Jenkins Pipeline job. This pipeline:

• Runs daily at 2 AM.
• Logs into source and target FME Flow environments.
• Downloads .fsproject files updated in the last 24 hours from the source environment.
• Uploads them to the target environment using FME Flow REST API (v4).

Prerequisites

To successfully execute this pipeline, the following prerequisites must be in place. Please be advised that this is a basic pipeline script that requires review for any security concerns. Any customization in the setup may require editing the Jenkins pipeline script.  A PDF document of the full pipeline script is attached to this article.

Jenkins Setup

• A Jenkins server with a build agent labelled “build-server-1” was set up. In our test, we are running a distributed Jenkins installation [master and worker setup] on Linux-based AWS EC2 instances. We also installed Nginx reverse proxy to access Jenkins on port 80 [Jenkins runs on port 8080 by default].
• Required plugins were installed on Jenkins:
  • Git plugin
  • Pipeline plugin
  • Terraform Plugin
  • Github Integration Plugin
  • Credentials Binding plugin
  • Pipeline Utility Steps Plugin
• The job type “pipeline” was configured.
• FME Flow CLI was installed at /usr/local/bin/fmeflow on the build server. Follow GitHub - safesoftware/fmeflow-cli: A Command Line Interface for interacting with FME Flow for more information on installing the CLI.
• Network access to the FME Flow instance must be provided.
• Jenkins credential entries were created for the source FME Flow (fmeflow-login-source) and target FME Flow credentials (fmeflow-login-target).

FME Flow Setup

• Both environments (Dev/Prod) were running on the same Flow version and have REST API v4 enabled.
• Flow user accounts logging in to run the CLI commands/API calls have been given access to the FME objects in the Projects. We logged in as a user with fmesuperuser role in this example.
• Please ensure that the encryption key is the same between Dev and Prod installs 

Pipeline Structure

Agent Declaration

pipeline {
    agent { label 'build-server-1' }
    ...}

The pipeline runs on the build-server-1 agent, where CLI tools and network access are preconfigured. It is triggered daily at approximately 2 AM using a CRON schedule:

triggers {
        cron('H 2 * * *')}

Users can select the following at runtime:

• FMEFLOW_SOURCE_ENV: Source environment (Dev or Prod)
• FMEFLOW_TARGET_ENV: Target environment (Dev or Prod)

Environment Block

environment {
    SOURCE_CONFIG = "${WORKSPACE}/source.yaml"
    TARGET_CONFIG = "${WORKSPACE}/target.yaml"
    PASSWORD_FILE = "${WORKSPACE}/fmeflow_pass.txt"}

These are used to store temporary configuration files and credentials for CLI operations.

• SOURCE_CONFIG: Where to save the YAML file that stores the API token generated after logging in to the source install.
• TARGET_CONFIG: Where to save the YAML file that stores the API token generated after logging in to the target install.
• PASSWORD_FILE: Where to temporarily store the admin login password (this is deleted later for security). In this example, the admin password is the same for the source and target install.

Resolve Environment URLs

def envMap = [
    'Dev'  : 'http://ec2-99-79-127-94.ca-central-1.compute.amazonaws.com',
    'Prod' : 'http://ec2-35-183-129-23.ca-central-1.compute.amazonaws.com'
]

This line defines a Groovy map (dictionary) named envMap. In this map:

• Dev is the key for the Development environment
• Prod is the key for the Production environment
• Each key maps to the corresponding FME Flow base URL for that environment

This converts the user-friendly environment labels into actual FME Flow URLs.

Log in to Source FME Flow

fmeflow login <SOURCE_URL> --user ... --password-file ... --config source.yaml

This uses Jenkins credentials to securely log in, generates source.yaml file with an API token, and removes the password file after login for security.

Find and Download Updated Projects

fmeflow projects --owner admin --output json --config ${env.SOURCE_CONFIG}

This retrieves all projects owned by the user admin in JSON format.

fmeflow projects download --name '${project.name}' -f '${project.name}.fsproject'

It then filters for projects updated in the last 24 hours, downloads each one as an .fsproject file and then writes project names to updatedProjects.txt.

Log in to Target FME Flow

This is the same as the step to log in to the source Flow install, but it generates target.yaml with an API token.

Upload Projects to Target Environment

For each .fsproject file, it uploads the file to the target FME Flow using REST API v4:

curl -X POST /fmeapiv4/projects/imports/upload

It then waits 15 seconds to allow import initialization, extracts the import ID from the Location header, then calls /run endpoint to import the uploaded projects:

curl -X POST /fmeapiv4/projects/imports/{ID}/run

Post Actions

post {
    success { echo ' Project sync completed successfully.' }
    failure { echo ' Project sync failed.' }
    cleanup {
        sh 'rm -f *.fsproject updatedProjects.txt ${SOURCE_CONFIG} ${TARGET_CONFIG} ${PASSWORD_FILE}'}}

This reports the pipeline result and always deletes temporary files after execution (the cleanup stage).

Conclusion 

This pipeline should automatically identify and sync the FME projects in both your environments every day.