Search

Scaling FME Server Dynamically on Microsoft Azure

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

Introduction

If you are using the Azure Distributed Windows deployment then it leverages Azure virtual machine scale sets. Scale sets provide both a management and automation layer to make it easier to run highly available and scalable applications on Azure.

In this article, we will review different options for scaling the FME Engine Scale Sets.

 

Licensing

For these scenarios, it makes sense to be using the new FME Server Dynamic Engines licensing. If you are using Standard Engines you will end up paying for capacity that you are not using. By licensing the FME Server with dynamic FME Engines, you can scale the FME Engines up and down and you will only consume credits when you run jobs.

 

Manual Scaling

The simplest way is to manually scale the number of FME Engine hosts up or down:

  1. In the Azure portal, select the `fmeserver-engine` Virtual Machine scale set.
  2. Click Scaling under Settings.
  3. You can then scale the number of hosts up or down using the slider.

Scaling on a Schedule

These instructions are based on this document here and detail how to scale up and down the FME Engine hosts.

  1. In the Azure portal, select the `fmeserver-engine` Virtual Machine scale set.
  2. Click Scaling under Settings.
  3. Choose Custom autoscale.
  4. Set the Scale mode to Scale to a specific instance count.
  5. Set the Instance Count to scale to the number of instances you want running when no scaling conditions are met.
  6. Add a scale condition and select Scale to a specific instance count
  7. You can then define your schedule.

In this example below, the scheduler will scale the number of FME Engine VMs from 1 to 5 Monday - Friday, during a two hour period in the afternoon. Since each instance has 2 FME Engines installed the capacity will increase from 2 FME Engines to 10 FME Engines. Since the FME Engines are pre-configured, they will connect to the Default queue automatically and start taking jobs.

Scaling based on Azure Metrics

Scaling based upon a schedule can work for specific workflows where the workloads are predictable. However, when the load is not predictable, scaling up and down based upon a metric is often the best fit.

On Azure, you can trigger a scale event based on over 35 metrics. Metrics can be broadly broken down into three categories: CPU, Memory, Disk and Network. What you base your scale on really depends on your specific workflows and what the bottleneck is. CPU is often the bottleneck so let’s scale based upon Percentage CPU.

  1. Select the fmeserver-engine Virtual Machine scale set.
  2. Click Scaling under Settings.
  3. Choose Custom autoscale.
  4. Set the Scale mode to Scale based on a metric
  5. Add a metric rule.

Scaling based on FME Server Metrics

Often with FME Server, the system metrics on CPU/Memory aren’t that useful as they only indicate what is currently running, not what is waiting to run. To get around this you can use a custom metric from FME Server to base our scaling on the length of queues. There is a wide range of possible scenarios and useful metrics, which highly depend on individual scenarios. Here is an proof of concept showcasing the scaling of VMSS based on the number of queued jobs: Scaling Exmaple

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.