FME Flow Hosted: How to Speed Up Workflows with the Temporary Disk

Introduction

If you are reading/writing large amounts of data on FME Flow (formerly FME Server), the read/write speed of the disk may be a bottleneck. This article provides an overview of leveraging the temporary disk on FME Flow Hosted (formerly FME Cloud) to store temporary data.
 

Temporary Disk

To be able to use the power of the temporary disks fully, we need to look into and understand three different concepts:

1. Which data should go to the temporary disk
2. When does FME use the temporary disk
3. The relation between the size and performance of the temporary disk

After understanding the implications and effects of these three aspects, we can exploit the temporary disk on FME Flow Hosted to give the workflows a performance boost.
 

Which data should go to the temporary disk?

On the primary disk, we store the FME Flow installation and the FME Flow Shared resources with one exception, the Temp folder. The Temp folder in the FME Flow Resources is connected to the Temporary Disk of your FME Flow Hosted instance.

The Temp folder should only be used for data that does not need to persist on your FME Flow Hosted instance. Data that does not need to persist is usually uploaded user data that is processed in one of your workspaces, data that you might download from a cloud resource like AWS S3 to process on FME Flow Hosted, and data FME needs to write to disk during a transformation (we'll look into more details about this soon).

There are a couple of reasons why this data should go to the temporary disk instead of the primary disk:

1. Stability: When the primary disk is full, the Web Application server might shut down and fail to start up correctly even after an instance reboot. Often, a rollback to a previous backup is the only way to recover. By storing data that is hard to control and estimated in size on the temporary disk, you reduce the risk of crashing the server if the disk fills up.
2. Flexibility: The primary disk can only be increased in size, while the temporary disk can be increased and decreased to match your requirements at all times.
3. Availability: Because the temporary disk is purged every time the FME Flow Hosted instance is rebooted, you can purge the whole disk easily, for example, with a scheduled restart on the weekend.
4. Affordability: The temporary disk is not included in the FME Flow Hosted backup and therefore, you can dramatically lower your backup costs by storing the data you don't need backing up on the temporary disk.

The relation between the size & performance of the disks

FME Flow Hosted runs on AWS EC2 instances and utilizes AWS General Purpose SSD (GP2) Volumes. Two important details to know about these volumes are the I/O Credits and the Burst Performance. Generally, the larger the disk, the higher the minimum input/output operations per second (IOPS).
 

Instance Launch

Resources

Please also check out our documentation and tutorials for more details on how to resize your temporary disk:

Documentation: Resizing an Instance