FME Flow Troubleshooting: Cleanup Tasks

Full Guide: FME Flow Troubleshooting Guide

Are you encountering issues creating, editing, or running FME Flow (formerly FME Server) Cleanup Tasks? Please read below for some common troubleshooting tips and resources.

In 2023, FME Server underwent a name change and is now known as FME Flow. Since this article discusses features present in previous versions of FME, it will refer to both names interchangeably, using the appropriate product name based on the year the feature was introduced. For more information on the rebranding, see our website. 
 

Content Overview

• Initial Troubleshooting
• Common Issues
  • “I’ve scheduled a task to clean up old files once a week, but the files are disappearing every day.”
  • “There are no tasks listed under System Configuration > System Cleanup > Tasks”
  • “My files are not being deleted even though I have a cleanup task scheduled.”
  • “The log file for Job ID <n> does not exist”
  • “Logs from FMEFlowJobSubmitter temporary engines are not being cleaned up”
• Are you still experiencing issues?
• Have ideas on how to improve this?

Initial Troubleshooting

• View the log file for Cleanup tasks in Resources > Logs > Service > current/old > fmecleanup.log and check for any errors
• If you have a distributed install make sure all the FME Flow Components (Web Application/Core/Engines/Database) are running and have the correct permissions.

Common Issues

“I’ve scheduled a task to clean up old files once a week, but the files are disappearing every day.”

Your FME Flow host is likely low on disk space. FME Flow will monitor disk conditions and if they go below a set threshold aggressive cleanup tasks are triggered. You should look into freeing up some disk space on your FME Flow host machine.
 
Alternatively, you can increase the threshold at which an aggressive cleanup is triggered via the FME Flow Web UI under System Configuration > System Cleanup > Configuration or disable low disk conditions entirely by setting these values to zero. WARNING! If your disk space is low, you should fix this, otherwise, other problems with FME Flow functions may start to occur.  

“There are no tasks listed under System Configuration > System Cleanup > Tasks”

As part of the installer a series of batch files are provided, one of which creates your cleanup tasks. In FME Flow (Server) 2018.1 and newer, the installer should run these scripts automatically as part of the installation, however, if the cleanup tasks or anything else is missing from the FME Flow menu you should manually run these scripts. 

1. Go to <FMEInstallDir>\Clients\utilities
2. Right-click on the AddCleanupTasks.bat (or .sh on Linux) and run as administrator
3. Restart FME Flow

If this script failed to run, it’s likely the others did as well, in this case, it is recommended you also run addPublishers, addSubscribers, addServices and configureDemoWorkspaces before restarting FME Flow. 

In FME Server 2018.0 and older the post-configuration scripts must be run manually after the installer is complete following these instructions.

“My files are not being deleted even though I have a cleanup task scheduled.”

• the task is enabled
• If there is a filter on the task do your files meet the criteria

“The log file for Job ID <n> does not exist”

There is a System Cleanup task to remove job logs, by default this removes log files when they are 1 week old. If you try to view a log that is older than this cleanup date, the log file will have been deleted by this cleanup task and this error will be displayed. If the job log you are trying to view was run in the last 7 days then your machine may have undergone an aggressive system cleanup task which is triggered in low disk conditions. Check out this article for more causes and resolutions. 

“Logs from FMEFlowJobSubmitter temporary engines are not being cleaned up”

When submitting jobs via the FMEFlowJobSubmitter transformer with the Submission Mode parameters are set to Submit Jobs = In Sequence and Wait For Jobs To Complete=Yes a child engine process is spun up to run those jobs. This engine takes the name of the parent engine with the prefix @<ParentJobID>.   

For example, if the job containing the FMEFlowJobSubmitter was running on Engine 1 with ID 123, the child job would run on engine <HOST>_Engine1@123. When these engines start up a log file is written to Resources > Logs > Engine > Current and is not handled by the system cleanup tasks. This can result in a large number of log files building up in this location. 

This issue has been resolved for FME 2022.0, where these log files are no longer created at all. To resolve this issue in an earlier version of FME: 

For both Windows and Linux

1. Run a Text Editor as administrator and open fmeEngineConfig.txt (located in <InstallDir>\Server)
2. Comment out, or remove the line starting with FME_SERVER_LOG_FILE

Note: This will suppress the log files from being created for all engine processes, not just those created by the FMEFlowJobSubmitter. If you would like to delete only the child engine logs, consider creating a workspace using the REST API and running it on a Schedule. See an example workspace attached. 

Are you still experiencing issues?

Please consider posting to the FME Community Q&A if you are still experiencing issues that are not addressed in this article. There are also different support channels available.

Have ideas on how to improve this?

You can add ideas or product suggestions to our Ideas Exchange.