Provisioning High Availability Flow Infrastructure on AWS using Jenkins

This article walks through a Jenkins pipeline that automates the provisioning and teardown of AWS infrastructure using Terraform. It provides parameterized control over plan, apply, and destroy actions for deploying FME Flow environments.

Prerequisites

The following prerequisites must be in place to successfully execute this pipeline. Please be advised that this is a basic pipeline script that requires review for any security concerns. Any customization in the setup may require editing the Jenkins pipeline script. A PDF document of the full pipeline script is attached to this article.

Jenkins Setup

• A Jenkins server with a build agent labelled “build-server-1” was set up. In our test, we are running a distributed Jenkins installation [master and worker setup] on Linux-based AWS EC2 instances. We also installed Nginx reverse proxy to access Jenkins on port 80 [Jenkins runs on port 8080 by default].
• Required plugins were installed on Jenkins:
  • Git plugin
  • Pipeline plugin
  • Terraform Plugin
  • Github Integration Plugin
  • Credentials Binding plugin
  • Pipeline Utility Steps Plugin
• The job type “pipeline” was configured.  

GitHub Repository

• A GitHub repository was created by forking Safe Software’s official IaC repository GitHub - safesoftware/fme-server-iac-templates.
• A Jenkins credential entry was created for access to the repo [using webhook and personal access token] to ensure Jenkins can access Git.
• Git was installed on the Build server

AWS Access and Credentials

• AWS CLI was installed on the Jenkins build server.
• AWS Access Key [aws-access-key] and Secret Key [aws-secret-key] for an AWS user account were created and stored as Jenkins credentials.
• Ensure the user has appropriate permissions to create, view and delete the AWS resources (e.g., EC2, VPC, IAM, RDS). 

Terraform

• Terraform was installed on the Jenkins build agent.
• The Terraform configuration files are located under the AWS/terraform folder in the IAC GitHub repository GitHub - safesoftware/fme-server-iac-templates. 

AMI Details

• AMI IDs for both FME Core and FME Engine must be available in the selected AWS region. These were created via the Jenkins job explained in  Building AMIs with Packer.

Pipeline Structure

Agent Declaration

pipeline {
    agent { label 'build-server-1' }

agent specifies which Jenkins node (agent) should run the job. In this case, it's pinned to a node labelled 'build-server-1' — ensuring the job runs on a known, configured executor with necessary tools (like Terraform and AWS CLI).

Environment Variables

environment {
    TF_WORKING_DIR = "AWS/terraform"
    TF_VERSION = ">=5.32.0"
    AWS_REGION = "ca-central-1"}

These variables define environment-specific and reusable values:

• TF_WORKING_DIR: Specifies the relative path where Terraform files are located.
• TF_VERSION: Declares the required Terraform version
• AWS_REGION: Specifies the AWS region in which resources will be created (Canada Central in this example).

Parameters Block

parameters {
    choice(name: 'ACTION', choices: ['plan', 'apply', 'destroy'], description: 'Choose Terraform Action')
    string(name: 'FME_CORE_IMAGE_ID', description: 'Enter the FME Core AMI ID')
    string(name: 'FME_ENGINE_IMAGE_ID', description: 'Enter the FME Engine AMI ID')}

These are dynamic inputs provided at runtime by the user:

• ACTION: Decides which Terraform command to run — useful for reusability.
• FME_CORE_IMAGE_ID / FME_ENGINE_IMAGE_ID: Allows passing different AMI IDs per deployment for core/engine images. We would provide the customized AMIs created via Packer in Building AMIs with Packer 

Checkout

stage('Checkout') {
    steps {
        git branch: 'main', 
            credentialsId: 'github-cred', 
            url: 'https://github.com/MerlineG-safeSoftware2024/fme-server-iac-templates.git'}}

This clones the IaC repo containing Terraform configurations:

• branch: Ensures mainline code in GitHub is used
• credentialsId: Uses Jenkins stored GitHub token (github-cred)

This enables controlled deployment based on code version.

Terraform Init

stage('Initialize Terraform') {
    ...
    sh 'terraform init -upgrade'}

This initializes Terraform and downloads required providers or modules. The -upgrade option ensures the latest versions are pulled (as long as compatible).

Terraform Validate

stage('Validate Terraform') {
    ...
    sh 'terraform validate'}

This stage validates the Terraform config to catch syntax or logic errors early, before attempting any state operations.

Terraform Plan

stage('Terraform Plan') {
    when {
        expression { params.ACTION == 'plan' || params.ACTION == 'apply' }}

This runs only if the user selects plan or apply. Inside the steps:

• AWS credentials are injected using the Jenkins credentials store.
• AWS CLI and Terraform environment variables are set via export.
• terraform plan:
  • Uses input variables:
    • fme_core_image_id, fme_engine_image_id: for deploying EC2 instances
    • ad_admin_pw, db_admin_pw: passwords for internal system configurations. You can either provide them as input at runtime or look into pulling them from the AWS password manager instead.
    • owner: tracks who deployed the installation.
  • Runs in dry-run mode, outputting a .tfplan file.

Terraform Apply

stage('Terraform Apply') {
    when {
        expression { params.ACTION == 'apply' }}

This executes if the user selects apply. It uses credentials to authenticate to AWS and applies the previously generated plan file tfplan using -auto-approve to skip confirmation.

Note: The plan must be executed in the same working folder and context to succeed.

Terraform Destroy

stage('Terraform Destroy') {
    when {
        expression { params.ACTION == 'destroy' } }

If the user selects Destroy, this destroys the infrastructure defined in the Terraform files. It uses the same injected credentials and variables as the plan. auto-approve skips manual prompt.

Post Block

post {
    always {
        echo "Pipeline execution completed"}}

This runs after every pipeline execution, regardless of success or failure. It is useful for consistent logging or cleanup.

Conclusion

If plan is selected, this pipeline will list all the AWS resources that will be created to set up a fault-tolerant installation. 

If apply is selected, this pipeline should output the ALB endpoint to access the High Availability Flow UI setup.  

Destroy will list all the resources to be destroyed and confirm the removal.