How to use the FME Flow Streams Interface

Introduction

In this tutorial, we will be going through the steps required to publish, create, and manage streaming workspaces in FME Flow. Deploying streams on FME Flow is a simple and straightforward process.

Step-by-step Instructions

Part 1: Publishing Stream Workspaces to FME Flow

Before publishing your workspace to FME Flow, make sure all transformers are configured correctly and that the workspace is set to run in Stream mode. For more information on how to design stream workflows, please read our FME Form authoring tips article.

Publishing a stream processing workspace to FME Flow is no different than any other workspace. Several stream source connectors will require an FME Package to run, so ensure that your user/role has permission to upload packages to FME Flow (unless the package has already been uploaded).

To run the stream workspace on FME Flow, register it with the Job Submitter service. Running a stream processing workspace is not to be confused with the Data Streaming service since that service is for generating data feeds from a single output file (i.e, returning JSON or HTML to a web client after the job completes).  Stream processing workflows run indefinitely, but Data Streaming workspaces are meant to finish at a certain point. 
If you are looking for a Data Streaming Service example, see Streaming XML with FME Flow. 

 

Part 2: Creating a Stream

Open your FME Flow web interface, and from the side navigation, click Streams > Create Stream.

Enter a name for your stream and optionally give it a description.  

Log Debug should be left off. If you encounter issues with your stream after it has been created, you can enable troubleshooting to resolve the problem.

Optional: Enable Remote Stream

If you want to run the stream on a remote engine to move your stream processing closer to the data source, then enable Remote Stream. Remote Engines Services must be configured first. To learn more, please see Leverage Remote Engine Service in FME Flow. After enabling Remote Stream, choose the Remote Engines Service you want to use. 

Next, select the repository and workspace you just published and fill in any published parameters. Click Create.

Part 3: Managing Stream Engines

After creation, the stream will be started by default, but no jobs will be running until at least one engine is assigned. Engines assigned to a stream are no longer available to other FME Flow jobs, so CPU-usage engines can be a good choice for streams that are inactive for large periods of the day. To learn more, see Getting Started with CPU-Usage (Dynamic) Engines.

Under Assigned Engines, click Manage. In the "Manage Engine Rule" pop-up window, choose to assign engines by Name or Property. 

If Type is Name, select one or more engines by name. Any FME Engines assigned to a stream will remove them from the queue they were assigned to and reserve them exclusively for running the stream. If you configured the stream as a Remote Stream in Part 2, then only the engines from the Remote Engines Service will be available for selection. 



If Type is Property, add rules and specify the properties to determine how engines are assigned to the stream. Select from the list or manually enter custom properties. Please see the FME Flow documentation on Queue Control for more information.  

A single engine may not be sufficient to process a high volume of incoming stream data. To handle these scenarios, you can scale up FME Flow’s stream processing capacity by assigning more than one engine to a stream. More information on the scalability of FME Flow stream processing is available in this article under Windowing Across Multiple FME Engines.

Once the engine(s) are assigned, the stream's job(s) will start. The engine(s) used by a stream can be modified at any time under the Assigned Engines section. Click Manage to reopen the "Manage Engine Rule" window. 

Part 4: Stream Jobs

When a stream is started, jobs are run/queued as soon as engines are assigned to the stream. If the assigned engine is in the middle of running a different job, the stream job will be queued until the engine is free.



View your stream job statuses under the Jobs section.  There are three filtering options you can apply:

• All: Queued and Running jobs.
• Running: Jobs that are currently being run by an engine.
• Queued: Jobs that are submitted but not yet picked up by an engine to run.

Part 5: Starting, Stopping, and Removing Streams

To start or stop a single stream, open the stream by clicking its name on the Manage Streams page, and then click Start or Stop.

To start, stop, or remove one or more streams, go to Manage Streams, select the streams, and open Actions. 

Part 6: Viewing Job Logs to Troubleshoot Streams

To troubleshoot your FME Flow stream processing workflows, view the job logs. Stream jobs are logged similarly to workspaces on FME Flow. 

Stream job logging can be monitored while the stream is actively running. You can view the log file of a currently running stream by clicking the job on the stream page. 

When viewing the log file, click the Logs resource folder link to view old logs. This is where you will find job logs from stream jobs that have automatically restarted due to failure or logs that have been split because they have gotten too large over time. 

You can also view the job log by navigating to the Running page under Jobs. If a stream was stopped or canceled, you can find the log under the Completed jobs, and if a stream is queued to run, you can find it under Queued jobs.

When viewing the job history, you can filter the list to view only stream jobs by opening Filters and setting Source Type to Streams.

If you are having issues with a streaming job, enabling Log Debug mode may help. This will produce a more verbose job log. Enable Log Debug mode by editing the stream, turning on Log Debug, and saving. After saving, the stream's job(s) will be restarted in Log Debug mode.  

Part 7: Managing Streams with the REST API

It’s possible to manage your streams through the FME Flow REST API.  The following examples illustrate what kinds of calls you can make.

Please note that these endpoints are part of the FME Flow REST API v4, which was officially released in FME 2025.1. 

{
    "name": "TestStream",
    "description": "My stream description.",
    "enabled": true,
    "request": {
        "workspace": "StreamWorkspace.fmw",
        "repository": "my_repo",
        "publishedParameters": {
            "my_parameter": "value"
        }
    }
}

{"enabled":true}   Set the value to 'false' to stop the stream.

{"engine":"<engine_name>"}

As of FME Flow 2025.1, the V3 REST API has been deprecated. The current API will remain accessible, but no new features or general bug fixes will be implemented. Starting with FME Flow 2026.1, the V3 API will be removed. We encourage all users to prepare their systems for this change. 

The V4 REST API offers enhanced security and usability, including new and reorganized endpoints, as well as the removal of certain endpoints. For information on migrating from V3 to V4, please see the FME Flow REST API V4 documentation and click on the heading Migrating from REST API V4.
 

Additional Resources

FME and Stream Processing
FME Form Tips for Working with Continuous Data Streams
Windowing Data Streams in FME
Webinar: Introduction to Data Stream Processing
Webinar: Empowering Real-Time Decision Making with Data Streaming