Introduction
This article will explore a technical overview of implementing CI/CD with FME, focusing on a practical deployment scenario using tools commonly used by users. We'll examine the essential components and the configurations that enable a successful continuous integration and deployment of FME.
Scenario
We will be looking at a scenario where a user wants to configure their FME Flow deployments with CI/CD using
- AWS as their cloud and deployment target
- Jenkins as their CI/CD platform
- GitHub as the central repository system
- Terraform as their Infrastructure as code tool
This also takes into account that the user has expertise in the above-named tools
Configuration Process
The overall process can be complex depending on the type of CI/CD flow you would like to implement; however, for a simple FME deployment, it usually involves these four general steps
- Source
- Build
- Test
- Deploy
The diagram below describes the flow
CI/CD lifecycle
In the context of this deployment with FME, here are that would be performed
1. Build Image
This is one of the most fundamental aspects of the overall FME CI/CD pipeline, as the image serves as the final FME deployment once the CI/CD workflow is completed. You will need to source the code needed to create a custom AMI(with FME installed) with Packer from the IAC repository. This process will be automated with Jenkins, as described in this article. The Jenkins setup requires a configured Jenkins server with all the necessary plugins and proper AWS credentials of the deployment target.
Key pipeline components include the agent declaration, parameter configuration, environment variables, and various operational stages. A real scenario with the actual process to build the AMI can be found here.
2. Set up and Trigger the Pipeline
This is the main step of the CI/CD process, as it deploys FME after preparing the created image. This process involves creating a structured Jenkins pipeline that automates the provisioning and teardown of AWS infrastructure using Terraform. The Jenkins server at this point should have the necessary plugin,s which include the Git, Pipeline, Terraform, GitHub Integration, Credentials Binding, and Pipeline Utility Steps plugin. You'll need to clone the IaC repository and add the GitHub credentials to Jenkins for secure GitHub access. After this, you can access and deploy all the deployment options available on the IAC template as long as the repository information is entered in Jenkins. The pipeline structure should follow a logical progression with clearly defined stages. An example of how this looks, along with a working full configuration, can be found in this article
3. Test the Deployment
This involves using Terraform to validate and display the resources the pipeline will deploy. The process to configure this stage is also found in this document. This stage typically helps some save, as there are some errors that would be caught in this stage without having to wait for the complete deployment
4. Run the Deployment
At this point, the pipeline is created using the pipeline structure and can be deployed to its target environment in AWS. In this stage, you can determine whether you want to deploy to a production or a dev environment. If any other deployment needs to be made after this stage, the process will start again from the first stage.
Once these steps are completed and you have a complete deployment, a user can trigger their FME deployment pipeline anytime, and they can just make changes to the IAC template based on their own situational needs. Once the pipeline is triggered, it follows the process described in the CI/CD lifecycle diagram above
Important Components to Consider in the Typical FME CI/CD Deployment
FME CLI
The FME Flow CLI is a command-line interface that uses the FME Flow REST API to interact with FME Flow through a set of intuitive commands and flags for various operations. The CLI enables the automation of FME operations in scripts and CI/CD pipelines, and is very valuable for automated testing scenarios where manual intervention isn't feasible. For information about the FME Flow CLI, including installing and running, see the FME Flow CLI page. The FME Flow CLI would have to be installed at this directory /usr/local/bin/fmeflow on the Jenkins build server to have its functionality available to use by Jenkins for the automated parts of the FME installation. Some of the commands it would run as part of the installation process can be found here.
FME API
This provides programmatic access to FME functionality, enabling integration with custom applications and automation tools in the CI/CD pipeline. The API offers comprehensive access to FME Flow's capabilities, including workspace management, job submission, scheduling, and resource handling.
Packer
Automates the creation of machine images, helping to create consistent FME Flow environments across development, testing, and production. Packer uses a declarative JSON configuration to define how images should be built, ensuring reproducibility across environments. It supports multiple platforms, including AWS AMIs, Docker containers, and VMware images, allowing teams to standardize their FME Flow deployments regardless of the target environment. Packer should also be installed on the Jenkins build server agent with the Packer template fme_flow_aws.pkr.hcl located here.
Github Repository
This stores the FME workspaces, scripts, and infrastructure code with version control, enabling collaboration between teams and change tracking. It also provides critical features for CI/CD, like pull requests for code review, issue tracking for defect management, and branch protection rules to enforce quality gates. Users can organize FME into logical repositories, separating concerns like infrastructure code, automation scripts, and FME workspaces.
Jenkins Plugins
Plugins are very important components for deploying a full CI/CD pipeline. Here are some of the commonly used Plugins and how they contribute to the deployment
Git Plugin
This plugin enables Jenkins to interact with Git repositories, pulling FME workspaces and configuration files for deployment. It provides integration with Git operations, supporting advanced features like checking out specific branches, tags, or commits as needed for deployment.
The plugin also handles authentication to private repositories that would likely be used in an organization through SSH keys or credentials stored securely in Jenkins. For FME deployments, this enables tracking deployment versions and fetching changes made to the IAC templates.
Pipeline Plugin
This basically allows the orchestration of the multi-stage process required to deploy FME Flow. The Jenkins Pipeline provides a domain-specific language (DSL) based on Groovy for defining deployment pipelines as code. For FME deployments, pipelines can define stages like workspace validation, unit testing, integration testing, staging deployment, and production deployment. The plugin supports both declarative and scripted pipeline syntax, offering the option to customize your FME deployment to any varying complexity.
Github Integration Plugin
This plugin enhances the connection between Jenkins and GitHub, enabling Webhook triggers and pull request integration for FME configuration changes. This plugin goes beyond basic Git operations by integrating with GitHub's collaborative features. It can automatically trigger builds when pull requests are created or updated, providing immediate feedback to developers about workspace validity and test results. The plugin can post build status updates back to GitHub, indicating whether changes are safe to merge. For FME deployments, this creates a powerful feedback loop where proposed changes are automatically validated before they're incorporated into the main branch. The plugin also supports branch filtering to limit which branches trigger builds, helping teams implement GitFlow or trunk-based development workflows for their FME components.
Credentials Binding Plugin:
This plugin securely manages authentication credentials for various services in the CI/CD pipeline, including AWS, FME Flow, and other systems that require them. This plugin also addresses the critical security concern of handling sensitive credentials in automation pipelines, which will be important for production environments or other sensitive deployments. It securely stores credentials in Jenkins' encrypted storage and makes them available as environment variables or files during pipeline execution without exposing them in logs or console output. For FME deployments, this enables secure authentication to AWS services, FME Flow, REST APIs, database connections, and other protected resources. It supports various credential types, including username/password pairs, SSH keys, AWS access keys, certificates, and secret text values.
Pipeline Utility Steps Plugin:
This plugin provides additional pipeline steps for working with files, JSON processing, and other utilities useful in FME deployment automation. This plugin extends Jenkins Pipeline with specialized steps for common automation tasks. For FME deployments, the JSON processing capabilities are particularly valuable for parsing and modifying FME Flow configuration files, API responses, and Terraform outputs. The plugin includes functions for reading and writing files, archiving and unarchiving content, comparing files, and finding files matching patterns. These capabilities enable complex deployment logic such as dynamically generating a configuration based on environment variables, extracting information from test results, or conditionally executing steps based on file content. The plugin also provides steps for SHA checksum verification, which can be used to validate FME deployment integrity
Best practice
When implementing a CI/CD pipeline for FME deployments, consider these best practices:
- Use version control for all infrastructure code and FME workspaces
- Implement proper secret management for credentials
- Create separate environments for development, testing, and production
- Automate testing before deployment to production
- Document the pipeline thoroughly for team knowledge sharing
- Implement monitoring for both infrastructure and FME components
Conclusion
If configured correctly, the overall time to deploy complex builds can be dramatically reduced. The approach described above is a high-level overview of the CI/CD pipeline lifecycle; however, users can customize their deployment flow using any approach that fits their organizational requirements, expertise, and tool availability.