Introduction
CPU-Usage Engines have the same processing capabilities as ‘Standard’ FME Engines.
However, instead of paying for a fixed license per engine, CPU-Usage Engines run on credits, giving you the option to purchase additional FME Flow processing power in a whole new way. One credit equals one hour of engine CPU time. From there, you only pay for the CPU time the engine spends running a job, or put another way, whilst the Engines are idle, you are not paying for anything!
CPU time is the key here. In many workflows, an engine might be running a job but spend a significant amount of its time waiting for data. With CPU-Usage Engines, you only pay for the CPU time, not the full runtime of the job. CPU time is composed of User (FME) and System (the OS performing work on FME's behalf) time.
You can identify the CPU time used by an engine at the end of the job log. For example, from the log snippet below, CPU time accounts for approximately 25% of the full job duration, and this is all that you would pay for.
FME Session Duration: 58.6 seconds. (CPU: 16.4s user, 4.5s system)Licensing of CPU-Usage Engines is “engineless”. By “engineless” we mean that when you apply a license file with CPU credits, you have the ability to use as many engines as your hardware can support. Alternatively, you can choose to stretch those credits further and only launch as many engines as needed.
This allows you to have any number of Engines available immediately for various data processing tasks – FME Flow administrators can adjust the number of CPU-Usage Engines at any time.
Video
This video was recorded in an older version of FME. The interface may look different, but the concepts remain the same.
Engine Type Definitions
FME Flow has 2 engine types; functionally, they are equivalent, but the pricing mode differs.
Standard Engines - This engine type has always been available on FME Flow and is licensed per engine.
CPU-Usage Engines - These engines are licensed by CPU time (called “credits”).
What are some use cases for CPU-Usage Engines?
It may be more financially viable to use CPU-Usage Engines rather than Standard Engines in some scenarios, given the flexibility they offer. Potential use cases include the following.
Peak Workflows
There may be times when demand on FME Flow exceeds available capacity. This forces organizations to spend valuable resources looking for ways to flatten the load to meet this demand.
Perhaps you receive unpredictable or infrequent high volumes of incoming data, and it may not be affordable to have a license for multiple Standard Engines, as they will, for the most part, not be fully utilized. That’s where CPU-Usage Engines come in, providing additional processing power immediately in a cost-effective manner.
Big Data Lift
Sometimes organizations need a big data lift from one system to another, which may require hundreds of engines. While this could be solved by purchasing additional engines, this is cost-prohibitive for a one-time or infrequent activity. Instead, purchasing Engine credits allows you to start as many CPU-usage Engines as needed.
FME Flow Apps
Flow Apps provide end users with a self-serve method to upload or download data. If an app is intended to return a result, a quick turnaround is important. However, if all your Engines are occupied or there are many queued jobs, the end user will have to wait.
This may become increasingly challenging as the Flow App gains popularity. By launching CPU-Usage Engines dedicated to processing Flow App requests, you can better manage load and eliminate potential wait time. All whilst ensuring you are optimizing your Standard Engine usage.
IoT/Data Streaming
If you have a workspace monitoring a streaming service, a dedicated engine is required to run continuously, ready to receive incoming traffic. However, it’s likely there will be quiet periods when this workspace isn't processing anything but is simply waiting to receive new data.
Using CPU-Usage Engines provides a cost-effective solution for monitoring this service even during periods of low activity.
CPU-Usage Engine FAQs
What are the requirements for using CPU-Usage Engines?
- An environment that supports FME Flow
- CPU-Usage Engine credits
- Automatic licensing to synchronize remaining credits with Safe Software. Please see the documentation for information on what URLs FME Flow communicates with for licensing.
How can I find out how much CPU time a workspace is using?
In FME Flow, this information can be found at the end of a job log:
FME Session Duration: 58.6 seconds. (CPU: 16.4s user, 4.5s system)In addition to the job logs, new job statistics, including CPU Time and CPU Percent, are available in the web UI. Job Statistics are viewable on the Jobs Completed, Workspace, Automations, and Schedule pages, and are accessible via the REST API. Check out Monitoring FME Flow Job Activity using the REST API for more information on how to use Job Statistics to create dashboards that can inform your decision-making.
As of FME Flow 2025.1, the V3 REST API has been deprecated. The current API will remain accessible, but no new features or general bug fixes will be implemented. Starting with FME Flow 2026.1, the V3 API will be removed. We encourage all users to prepare their systems for this change.
The V4 REST API offers enhanced security and usability, including new and reorganized endpoints, as well as the removal of certain endpoints. For information on migrating from V3 to V4, please see the FME Flow REST API V4 documentation and click on the heading Migrating from REST API V4.
You may observe discrepancies between the CPU time recorded in an FME Form/Flow job log and the time presented through Job Statistics (on the Jobs Completed page or via the REST API). This is expected because the time reported by Job Statistics includes the additional start and end scripting that FME Form/Flow logging processes cannot capture. The time recorded by Job Statistics is the more accurate total processing time, and this is the numeric value against which CPU-Usage Engines will be charged.
How can I control what jobs run on my CPU-Usage Engines?
In FME Flow, you can use the Queue Control functionality to create Queues, Job Routing Rules, and Engine Assignment Rules to control jobs. You can create Job Routing Rules based on Job Statistics to route jobs to a specific queue depending on their CPU time, percent, or other properties. Then create Engine Assignment Rules based on engine and queue properties. A single engine can be assigned to multiple queues, and multiple engines to a single queue.
The FME Flow Rest API /metrics Endpoint retrieves the number of Jobs currently waiting in each queue. An FME Flow Automation could be built around this endpoint to reassign Engines to different queues based on the current job load. Take a look at the videos at the bottom of this article for an example using this endpoint.
How can I change the number of CPU-Usage Engines?
FME Flow Administrators can change the number of available CPU-Usage Engines at any time, with no restriction on the number.
You can start new engines by changing the count on the Engine Management page of the FME Flow Web UI. You will need Administrator privileges to access this page. Remember, you can start as many engines as you like, as you’ll only be paying for the time they are running jobs!
To scale engines automatically, you can use the FME Flow REST API V4 endpoint POST /fmeapiv4/enginehosts/{name}/engines/scale, where the name is the engine hostname, and the upload body is:
{
"numEngines": 2,
"type": "dynamic"
}If your FME Flow is running over Docker or Kubernetes, this option is not available in the Web UI.
For Docker, we include a commented-out block in the docker-compose.yaml file. Please locate this block “--- Uncomment to deploy CPU-Usage Engines with FME Flow ---” and uncomment the full block.
Then run the command below on the command prompt to set the number of CPU-Usage engines:
docker compose -f "<docker-compose.yaml filepath>" up --scale fmeflowenginedynamic=<number of engines> -dHow can I see if my job ran on a CPU-Usage Engine?
When you launch a CPU-Usage Engine, it is named <hostname>_Dynamic_Engine<n>, whereas Standard Engines are named <hostname>_Engine<n>. This allows you to easily identify the engine type on the Jobs Running/Completed page.
What are the hardware requirements?
Functionally, CPU-Usage Engines work in exactly the same way as Standard Engines. Therefore, whilst you can launch an unlimited number of additional CPU-Usage Engines, you should consider your operating system's hardware. Make sure there are enough physical or virtual resources (CPU, RAM) to support processing demand, which varies across workflows/data translations. For optimal performance, we recommend that each engine have a dedicated core.
Can I have CPU-Usage Engines spread across different host machines?
Yes, in exactly the same way as with Standard Engines, you can add distributed CPU-Usage Engines to your FME Flow architecture. See our documentation for step-by-step instructions to set this up.