Introduction
Adding 3rd-party components (for example, Esri Software) to an FME Flow deployment in the cloud usually requires manual installation on VMs and introduces limitations regarding the scaling capabilities. To overcome this, images containing all required components, including FME Flow and 3rd-party software, can be built and deployed programmatically to create a reliable, reproducible environment. In this article, we will discuss the approach Safe Software uses to build the images used in the Azure Marketplace Deployment, and introduce example templates that can be used to create custom images with 3rd-party software installed in Azure and AWS.
Hashicorp Packer
There are several different options for building VM images. Packer is an open-source tool for creating identical machine images for multiple platforms from a single source configuration. To get an overview of how Packer works and what functionality it offers, review their documentation.
In our example, we will only use Packer for building Azure Windows Images or AWS AMIs, to be later used in Terraform, Bicep, or CloudFormation deployment scripts.
Basic Concept to Build an FME Flow Image
In the simplest approach, the build template needs a source block and a build block. The source block describes an existing image as the base image, the specifications (size, virtual machine type, etc.) and metadata (name, tags, etc.) of the new image. The build block consists of multiple provisioners (in the case of the FME Flow example), which run installation scripts and copy configuration scripts to the image that run once a new VM is started from the image.
Install script
The second step of the build process is to run the FME Flow installation script (FME Flow Core & FME Flow Engine). These scripts are running during the build process and install FME Flow Core/FME Flow Engine in a generic way using placeholder parameters in a silent FME Flow install command.
Configuration Script (confd)
The configuration scripts (FME Flow Core & FME Flow Engine) are copied to the VM image in the first step of the build process. The parameters required by the scripts are dependent on the infrastructure of the deployment and will be passed in during the execution by the IaC tool once the image is deployed (in Azure Virtual Machine Scale Set or an AWS Auto-Scaling Group). The scripts use the configuration management tool confd to update the FME Flow configuration to match the infrastructure (e.g. fileshare, database, public & private hostnames). Each FME Flow installation ships with a confd executable and confd template files, allowing a programmatic configuration of FME Flow.
Sysprep
The last step in the image build process is to run a sysprep command, which prepares the Windows operating system to be used as an image by removing unique elements from the system (e.g. computer name, etc.). These commands might be different for each cloud service provider.
The same build process is used to create the public VM images used in Azure Marketplace deployments. For more details, review the example packer templates for Azure Windows VMs or AWS AMIs and the corresponding documentation for AWS and Azure.
Adding 3rd Party Components
FME Flow Engine vs. FME Flow Core Image
For a distributed deployment of FME Flow, two different images are created to allow independent scaling and support high availability and fault tolerance. When it comes to adding 3rd-party components, it is important to understand where to add additional components.
FME Flow Engine
Any 3rd party component that is required for specific format access (e.g., ESRI Geodatabase) or a specific library to be called via an FME Transformer (for example, Python) needs to be added to the FME Flow Engine image. The FME Flow Engine runs the jobs, so any 3rd-party components or dependencies that are needed to successfully run the workspace will need to be added to the FME Flow Engine image to make sure the same workspace will run successfully.
FME Flow Core
The FME Flow Core image includes the FME Flow Core and the Web Application Server. Any 3rd-party component related to web server functionality (e.g., nginx reverse proxy) should be added to the FME Flow Core image.
3rd Party Component Checklist
Before adding a 3rd party component to an FME Flow image, start by reviewing the following checklist:
Architecture & Networking
Does the 3rd-party software need specific ports opened (inbound or outbound) that aren't part of the standard FME Flow profile?
Does the 3rd-party component require SSL/TLS certificates to communicate with an internal database or API? If so, how are those injected into the image (e.g., via Azure Key Vault or AWS Secrets Manager)?
Does the software install for the current user or all users?
Versioning & Reproducibility
Are you installing the "latest" version or a specific version? To ensure the "reproducible environment", you must pin the 3rd party software version.
Licensing
Does the component need a license?
Can the license be provided as a file?
Is a connection to a license server needed?
Can the licensing be done via an API?
Installation & Configuration
Is the component a single file?
Does the component require installation?
Is a silent installation supported?
Does the component require configuration after installation?
Can the configuration be automated?
Does the component require specific entries in the System Environment Variables?
Build Process
Where is the .msi or .exe stored during the Packer build? Is there a secure S3 bucket or Azure Blob storage that the build agent can access?
Does the build script include a "smoke test"? (e.g., a command-line call to the 3rd-party tool to ensure it returns a version or success code before the image is finalized).
Runtime & Optimization
Does the 3rd party component log to a specific folder? You may need to redirect those logs to a centralized location (like CloudWatch or Azure Monitor), so they aren't lost when an Engine VM is terminated.
Does the 3rd-party installer leave behind large extracted temp files? Purge installer caches to keep the machine image small, which speeds up scaling.
Ideally, the image-building process is completely automated. Therefore, it is important to plan and have a clear understanding of the final configuration. If the 3rd-party component is just a single executable that can be called by the FME Flow Engine via a workspace, and a simple File Provisioner can be used to upload the file to the image.
For more complex applications, an installation might be required. In this case, it is recommended to create a PowerShell script to perform a silent installation using a PowerShell Provisioner.
Additional Resources
Please consider posting to the FME Community Q&A if you have further questions or issues that are not addressed in this article. There are also different support channels available.
The FME Flow documentation has some guidance on setting up load balancers.
You may also find it useful to browse other resources or forum questions regarding load balancers in case this addresses our concerns.