How to use the FME Flow Streams Interface

Introduction

In this tutorial, we will be going through the steps required to publish, create, and manage streaming workspaces in FME Flow (formerly FME Server).  Deploying streams on FME Flow is a simple and straightforward process.

Step-by-step Instructions

Part 1: Publishing stream workspaces to FME Flow

Before publishing your workspace to FME Flow, make sure all transformers are configured correctly and that the workspace is set to run in Stream mode.  For more information on how to design stream workflows, please read our FME Form authoring tips article.

Publishing a stream processing workspace to FME Flow is no different than any other workspace.  Several stream source connectors will require an FME Package to run, so ensure that your user/role has permission to upload packages to FME Flow (unless the package has already been uploaded).

To run the stream workspace on FME Flow, register it with the Job Submitter service.  Running a stream processing workspace is not to be confused with the Data Streaming service since that service is for generating data feeds from a single output file (i.e: returning JSON or HTML to a web client after the job completes).  Stream processing workflows run indefinitely, but Data Streaming workspaces are meant to finish at a certain point. 
If you are looking for a Data Streaming Service example, see Streaming XML with FME Flow. 

 

Part 2: Creating a stream

Open your FME Flow web interface, and in the side navigation, click Streams > Create Streams.  Assign a name for your stream and optionally give it a description.  Next, select the workspace from the repository you just published and fill in any published parameters.  Click OK when done.

 

 

Part 3: Managing stream engines

The next page is where you can configure additional settings for your stream.  The stream will be started by default, but no jobs will be processed until at least one engine is assigned.  Under Assigned Engines, click Manage.  In the "Manage Engine Rule" pop-up window, choose to assign engines by Name or Property*. 

If Type is Name, select one or more engines by name.  Any FME Engines assigned to a stream will remove them from the queue they were assigned to and reserve them exclusively for running the stream.



If Type is Property, use boolean logic to specify the properties that determine how engines are assigned to the stream. Select from the list or manually enter custom properties. Please see FME Flow documentation on Queue Control for more information.  



A single engine may not be sufficient to process a high volume of incoming stream data.  To handle these scenarios, you can scale up FME Flow’s stream processing capacity by assigning more than one engine to a stream.  More information on the scalability of FME Flow stream processing is available in this article under Windowing Across Multiple FME Engines.
 
The engine(s) used by a stream can be modified at any time under the Assigned Engines section. Add engines by clicking Manage to reopen the "Manage Engine Rule" window. 



When using Name as an Engine Rule, remove engines from the stream by deselecting engines from the list. All deselected engines will be returned to the Default queue.

When using Property as an Engine Rule, modify the boolean logic to include or remove engines from the stream. 
 

Part 4: Stream jobs

To start or start a single stream, you can go to the stream’s page and click Start or Stop.  To start/stop one or more streams at the same time, go to ‘Manage Streams’, select the streams, and open Actions > Start/Stop.  To change a stream’s name, description, or workspace, click Edit.
 
When a stream is started, new jobs are run/queued as soon as engines are assigned to the stream.  Processing and logging will all go under the job ID listed in the table. If the assigned engine is in the middle of running a different job, the stream job will be queued until the engine is free.



View your stream job statuses under the Jobs section.  There are three filtering options you can apply:

• All: Queued and Running jobs.
• Running: Jobs that are currently being run by an engine.
• Queued: Jobs that are submitted but not yet picked up by an engine to run.

Part 5: Starting, stopping, and removing streams

To start or stop an individual stream, on the top of the stream’s page, you can click Start/Stop.



From the Manage Streams page, you can select one or more streams, and click Actions > Start/Stop/Remove.
 

Part 6: Debugging streams

To troubleshoot your FME Flow stream processing workflows, you may need to view the log files.  Stream jobs are logged the same as other workspaces on FME Flow.
 
Stream job logging can be monitored while the stream is actively running.  You can view the log file of a currently running stream by clicking the job on the stream page or navigating to the Running page under Jobs.  If a stream was stopped or canceled, you can find the log under the Completed jobs, and if a stream is queued to run, you can find it under Queued jobs.

When viewing the job history, you can filter the list to just stream jobs by opening Filters and setting Source Type to Streams.
 

Part 7: Managing streams with the REST API

It’s possible to manage your streams through the FME Flow REST API.  The following examples illustrate what kinds of calls you can make.

Please note that these endpoints are part of the FME Flow REST API v4, which has not yet been officially released as of November 2023. The endpoints are subject to change at any time.
 

{
    "name": "TestStream",
    "description": "My stream description.",
    "enabled": true,
    "request": {
        "workspace": "StreamWorkspace.fmw",
        "repository": "my_repo",
        "publishedParameters": {
            "my_parameter": "value"
        }
    }
}

{"enabled":true}   Set the value to 'false' to stop the stream.

{"engine":"<engine_name>"}

Additional Resources

FME and Stream Processing
FME Form Tips for Working with Continuous Data Streams
Windowing Data Streams in FME
Webinar: Introduction to Data Stream Processing
Webinar: Empowering Real-Time Decision Making with Data Streaming