Files
Introduction
In any active FME Flow environment, maintaining a clean and efficient server is a common challenge. Unused or forgotten Workspaces can accumulate, leading to clutter and increased management overhead. The FME Flow V4 API can be used to identify Workspaces in FME Flow that have never been run, as well as other associated FME Flow items such as Automations or Schedules. These unused items are prime candidates for deletion, a crucial step in keeping your FME Flow instance healthy and performant.
In this example, we will use the OpenAPICaller in FME Form to create an HTML report of Workspaces in FME Flow that have never been run, along with any Automations that include those Workspaces. This example could be expanded upon to include other related FME Flow items such as Schedules, Flow Apps, or Streams, using the appropriate V4 API endpoint.
Requirements
- FME 2024.1 or greater is required to access the OpenAPICaller transformer.
- For a detailed guide on using this transformer, please see: Getting Started with the OpenAPICaller.
The HTTPCaller could be used as an alternative to the OpenAPICaller for these tasks.
Ready-to-Run Workspace
This workspace is fully built and can be run by adjusting the user parameters below. To learn how it was created or how to extend it, see the step-by-step instructions.
- FME Flow Connection
- DestDataset_HTML
Step-by-Step Instructions
The Workspace is divided into three distinct parts: retrieving Workspace statistics, identifying Automation dependencies, and generating the final report.
Part 1: Get all the Workspaces
The first section of the Workspace is responsible for listing every Workspace on the server and checking its run history.
1. Retrieve Repositories
Because Workspaces are stored in repositories, the process begins here.
An OpenAPICaller is used to call the GET /repositories endpoint, retrieving a list of all Repositories.
A JSONFragmenter parses the response. The attributes are prefixed with repository_ (e.g., repository_name, repository_owner) to prevent confusion with other attributes later in the workflow.
2. Retrieve Items Per Repository
For every Repository found, the Workspace retrieves the contents.
A second OpenAPICaller calls the GET /repositories/{repository_name}/items endpoint. The repository path variable is set to the repository_name attribute from the previous step.
Since a Repository can contain various item types, such as Workspaces, Custom Formats, or Templates, the query is filtered to return only Workspaces by setting the Query String Parameter type to workspace.
3. Fragment and Tidy Attributes
A JSONFragmenter breaks down the response to get all Workspace items. Attributes are exposed with the prefix workspace_, specifically workspace_name and workspace_totalRuns.
An AttributeKeeper tidies the data, retaining only the essential attributes: repository_name, repository_owner, workspace_name, and workspace_totalRuns.
Part 2: Get All the Automation Dependencies
In parallel, the Workspace queries FME Flow to understand which Workspaces are currently being used in Automations.
1. Retrieve Automations
Similar to the Repository workflow, this stream starts by getting a list of all Automations.
An OpenAPICaller calls the GET /automations endpoint.
A JSONFragmenter parses the results, prefixing the attributes with automation_ and exposing automation_name and automation_id. Other attributes like automation_owner or automation_enabled status are available here if further filtering is needed.
2. Retrieve Dependencies
To see which Workspaces are in use, the Workspace requests the dependencies for each Automation.
An OpenAPICaller calls the GET /dependencies endpoint. The query parameters are configured with id set to the automation_id attribute, and type set to automation.
3. Process Dependency Lists
An Automation could have multiple dependencies returned as a list.
A JSONFragmenter is used to expose attributes dependencies{}.name and dependencies{}.type.
A ListElementCounter counts the number of dependencies each feature has, and a Cloner creates duplicate features according to this count. A ListIndexer then retrieves the dependency name and dependency type at the relevant index.
An earlier version of this article used a ListExploder instead of these three
transformers. This could cause dependency names and types to be overwritten by
dependency list values, resulting in some dependencies being lost from the process.
A ListElementCounter, Cloner, and ListIndexer ensure that all dependencies are
carried through to the next stage.
4. Filter for Workspaces
An AttributeManager renames the generic name and type attributes to dependency_name and dependency_type for clarity. Since dependencies can be various types, the data is filtered to retain only those where dependency_type = workspace.
Part 3: Join Automations to Workspaces, Filter no-runs, and Write to HTML Report
The final section merges the two data streams to produce a report.
1. Merge and Filter
A FeatureJoiner merges the data, joining workspace_name from Part 1 with dependency_name from Part 2.
To ensure all Workspaces are accounted for, even those not belonging to an Automation, the workflow uses both the Joined and UnjoinedLeft outputs.
A Tester filters this combined list, checking for workspace_totalRuns = 0. This isolates the Workspaces that have never been run.
2. Generate Report
An HTMLReportGenerator creates a table listing the identified Workspaces and their details.
Finally, an HTML writer writes the output to an HTML file, providing a clear audit of unused content on your FME Flow instance.
Results
The HTML report created by this Workspace lists all Workspaces in FME Flow that have never run, along with any related Automations.
This example process could easily be adapted and expanded to identify other items within FME Flow that are related to Workspaces that have never run, such as Schedules, Flow Apps, and Streams, using the appropriate V4 API endpoint.
Additionally, this Workspace itself can be hosted and run within FME Flow. This is a great candidate for use with a Dashboard, where the resulting HTML report can be written to the Dashboards folder, and viewed under the Dashboard tab of the Jobs page in FME Flow.