Deploy a distributed FME Server in the cloud with 3rd party components

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2022.0

Introduction

Adding 3rd party components (e.g, Esri Software) to an FME Server deployment in the cloud usually requires manual installation on VMs and introduces limitations regarding the scaling capabilities. To overcome this, images with all needed components, including FME Server and 3rd party software, can be built and deployed in a programmatic way to achieve a reliable and reproducible environment. In this article, we will discuss the approach that Safe Software uses to build the images used in the Azure Marketplace Deployment and introduce example templates that can be used to create custom images with 3rd party software installed in Azure and AWS.
 

Hashicorp Packer

There are several different options for building VM images. Packer is an open-source tool for creating identical machine images for multiple platforms from a single source configuration. To get an overview of how packer works and what functionality is offered, review their documentation.

In our example, we will only use Packer for building Azure Windows Images or AWS AMIs to be used in terraform, bicep or cloudformation deployment scripts.

 

Basic concept to build a FME Server image

In the simplest approach the build template needs a source block and the build block. The source block describes an existing image as base, the specifications (size, virtual machine type, etc.) and metadata (name, tags, etc.) of the new image. The build block consists, in the FME Server example, of multiple provisioners which run installation scripts and copy configuration scripts to the image that run once a new VM is started from the image.
 

Install script

The second step of the build process is to run the FME Server installation script (FME Server Core & FME Server Engine). These scripts are running during the build process and install FME Server Core/FME Server Engine in a generic way using placeholder parameters in a silent FME Server install command. 
 

Configuration script (confd)

The configuration scripts (FME Server Core ​​​​​​& FME Server Engine) are copied to the VM image in the first step of the build process. The parameters required by the scripts are dependent on the infrastructure of the deployment and will be passed in during the execution by the IaC tool once the image is deployed (in Azure Virtual Machine Scale Set or an AWS Auto-Scaling Group). The scripts use the configuration management tool confd to update the FME Server configuration to match the infrastructure (e.g. fileshare, database, public & private hostnames). Each FME Server installation ships with a confd executable and confd template files allowing a programmatic configuration of FME Server.
 

Sysprep

The last step in the image build process is to run a sysprep command which prepares the Windows operating system to be used as an image by removing unique elements from the system (e.g. computer name, etc.). These commands might be different for each cloud service provider.

The same build process is used to create the public VM images used in Azure Marketplace deployments. For more details, review the example packer templates for Azure Windows VMs or AWS AMIs and the corresponding documentation for AWS and Azure.
 

Adding 3rd party components

FME Server Engine vs. FME Server Core image

For a distributed deployment of FME Server two different images are created to allow independent scaling and support high availability and fault tolerance. When it comes to adding 3rd party components it is important to understand where to add additional components.
 

FME Server Engine

Any 3rd party component that is required for specific format access (e.g ESRI Geodatabase) or a specific library to be called via a FME Transformer (e.g python) needs to be added to the FME Server Engine image. The FME Server Engine runs the jobs so any 3rd party components or dependencies that are needed to successfullt run the workspace will need to be added to the FME Server Engine image to make sure the same workspace will run successfully.
 

FME Server Core

The FME Server Core image includes the FME Server Core and the Web Application Server. Any 3rd party component related to web server functionality (e.g nginx reverse proxy) should be added to the FME Server Core image.
 

3rd party component checklist

Before adding a 3rd party component to an FME Server image, start by reviewing the  following checklist:

  • Is the component a single file?
  • Does the component require installation?
    • Is a silent installation supported?
  • Does the component need a license?
    • Can the license be provided as a file?
    • Is a connection to a license server needed?
    • Can the licensing be done via an API?
  • Does the component require configuration after installation?
    • Can the configuration be automated?


Ideally, the image-building process is completely automated. Therefore it is important to plan ahead and have a clear understanding of the final configuration. If the 3rd party component is just a single executable that can be called by the FME Server Engine via a workspace and simple File Provisioner can be used to upload the file to the image.

For more complex applications, an installation might be required. In this case, it is recommended to create a powershell script to perform a silent installation using a PowerShell Provisioner.
 

Additional Resources

Please consider posting to the FME Community Q&A if you have further questions or issues that are not addressed in this article. There are also different support channels available.

The FME Server documentation has some guidance on setting up load balancers.
You may also find it useful to browse other resources or forum questions regarding load balancers in case this addresses our concerns.

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.