Getting Started with CPU-Usage (Dynamic) Engines

Introduction

CPU-Usage Engines (formerly Dynamic Engines) have the same processing capabilities as ‘Standard’ FME Engines.

However, as opposed to paying for a fixed license per engine, CPU-Usage Engines run on credits giving you the option to purchase additional FME Flow (formerly FME Server) processing power in a whole new way. One credit equals one hour of engine CPU time. From there, you only pay for the CPU time the engine spends running a job, or put another way, whilst the Engines are idle you are not paying for anything!

CPU time is the key here, in many workflows an engine might be running a job, but spend a significant amount of its time waiting for data. With CPU-Usage Engines you only pay for the CPU time, not the full runtime of the job. CPU time is made up of User (FME doing work) and System (the OS doing work on FME's behalf) time.

You can identify the CPU time used by an engine at the end of the job log. For example from the log snippet below, CPU time accounts for approximately 25% of the full job duration, and this is all that you would pay for.

FME Session Duration: 58.6 seconds. (CPU: 16.4s user, 4.5s system)

Licensing of CPU-Usage Engines is “engineless”. By “engineless” we mean that no aspect of the license has anything to do with the number of “engines” available. You are able to launch as many or as few engines as you want when you want.

This allows you to have any number of Engines available immediately for various data processing tasks – FME Flow administrators can adjust the number of CPU-Usage Engines at any time.

 

Video

 

Engine Type Definitions

FME Flow has 2 engine types, functionally they are equivalent but the pricing mode differs.

Standard Engines - This engine type has always existed on FME Flow, and is licensed per engine. 

CPU-Usage Engines - These engines are licensed by CPU time (called “credits”). 

 

What are some use cases for CPU-Usage Engines?

It may be more financially viable to use CPU-Usage Engines as opposed to Standard Engines in some scenarios, because of the flexibility they provide. Potential use cases include those listed below.

 

Peak Workflows

There may be times when the demand on FME Flow exceeds the capacity available. This forces organizations to spend valuable resources looking for ways to flatten the load to meet this demand.

Perhaps you receive unpredictable, or infrequent high volumes of incoming data, it may not be affordable to have a license for multiple Standard Engines, as for the most part they will not be fully utilized. That’s where CPU-Usage Engines come in, providing additional processing power immediately in a cost-effective manner.

 

Big Data Lift

Sometimes organizations have a need for a big data lift from one system to another which may require hundreds of engines. While this could be solved by purchasing additional engines this is cost prohibitive for a one-time or infrequent activity. Instead, purchasing Engine credits allows you to start as many CPU-usage Engines as needed.

 

FME Flow Apps

Flow Apps provide a method of self-serve to end users to upload or download data. If an app is intended to return a result of some kind a quick turnaround is important. However, if all of your Engines are occupied, or there are a number of queued jobs the end user will have to wait.

This may become increasingly challenging if the Flow App grows in popularity. By launching CPU-Usage Engines that are dedicated to processing Flow App requests you can better manage the load and remove any potential wait time. All whilst ensuring you are optimizing your Standard Engine usage.

 

IoT/Data Streaming

If you have a workspace monitoring a streaming service, a dedicated engine is required for this to be run continuously ready to receive incoming traffic. However, it’s likely that there will be quiet periods where this workspace may not be processing anything but simply waiting to receive new data.

Making use of CPU-Usage Engines provides a cost-effective solution to monitor this service even during periods of low activity.

 

CPU-Usage Engine FAQs

What are the requirements for using CPU-Usage Engines?

• An environment that supports FME Flow 2020.0 or newer

• CPU-Usage Engine credits

• Automatic licensing to synchronize remaining credits with Safe Software. Please see the documentation for information on what URLs FME Flow communicates with for licensing.

How can I find out the amount of CPU time a workspace is using? 

In FME Flow this information can be found at the end of a job log: 

FME Session Duration: 58.6 seconds. (CPU: 16.4s user, 4.5s system)

In addition to the job logs, new jobs statistics including CPU Time and CPU Percent are available in the web UI. Job Statistics are viewable on the Jobs Completed, Workspace, Automations, and Schedule pages as well as being accessible via the REST API. Check out Monitoring FME Flow Job Activity using the REST API for more information on how to use Job Statistics to create dashboards that can inform your decision-making. 

You may observe discrepancies between the CPU time recorded in an FME Form/Flow job log and the time presented through Job Statistics (on the Jobs Completed page or via the REST API). This is expected because the time reported by Job Statistics includes the additional start and end scripting that cannot be captured by FME Form/Flow logging processes. The time recorded by Job Statistics is the more accurate total processing time and this is the numeric value that CPU-Usage Engines will be charged against. 

How can I control what jobs run on my CPU-Usage Engines?

In FME Flow, you can use the Queue Control functionality to create Queues, Job Routing Rules, and Engine Assignment Rules to control jobs.  You can create Job Routing Rules based on Job Statistics to route jobs to a specific queue depending on their CPU time or percent or other properties. Then create Engine Assignment Rules based on engine and queue properties. A single engine can be assigned to multiple queues and multiple engines to a single queue.

The FME Flow Rest API /metrics Endpoint retrieves the number of Jobs currently waiting in each queue. An FME Flow Automation could be built around this endpoint to reassign Engines to different queues based on the current job load. Take a look at the videos at the bottom of this article for an example using this endpoint.

How can I change the number of CPU-Usage Engines?

FME Flow Administrators can change the number of available CPU-Usage Engines at any time, with no restriction on the number.

You can start new engines by changing the count on the Engine Management page of the FME Flow Web UI. You will need Administrator privileges to access this page. Remember, you can start as many engines as you like, as you’ll only be paying for the time they are running jobs!

To scale engines automatically you can use the FME Flow REST API V4 endpoint POST ​/fmeapiv4​/enginehosts​/{name}​/engines​/scale where the name is the engine hostname and the upload body is: 

{
  "numEngines": 2,
  "type": "dynamic"
}

Note: This endpoint is available in FME Flow 2021.0 and newer. REST API v4 is under tech preview, meaning these endpoints are still subject to change.

If your FME Flow is running over Docker or Kubernetes this option is not available in the Web UI, instead, you will need to scale up the number of Engines via the command line using:

docker-compose up --scale fmeserverengine=(number of engines)

How can I see if my job ran on a CPU-Usage Engine?

When you launch a CPU-Usage Engine it is given the name <hostname>_Dynamic_Engine<n> whereas Standard Engines are called <hostname>_Engine<n>. This allows you to easily identify from the Jobs Running/Completed page what engine type was used.

What are the hardware requirements?

Functionally, CPU-Usage Engines work in exactly the same way as Standard Engines. Therefore whilst you can launch an unlimited number of additional CPU-Usage Engines, you should take into consideration your operating systems hardware. Make sure there are enough physical or virtual resources (CPU, RAM) to support processing demand – which will vary between workflows/data translations. For optimal performance, we recommend that each engine have a dedicated core.

Can I have CPU-Usage Engines spread across different host machines?

Yes, in exactly the same way as with Standard Engines, you can add distributed CPU-Usage Engines to your FME Flow architecture, see our documentation for step-by-step instructions to set this up.

 

Additional Resources

FME World Tour 2022 Agile Resource Allocation Demo 
The Power of Dynamic Engines (blog)