FME Flow Troubleshooting: Cleanup Tasks

Full Guide: FME Flow Troubleshooting Guide

Are you encountering issues creating, editing, or running FME Flow (formerly FME Server) Cleanup Tasks? Please read below for some common troubleshooting tips and resources.

In 2023, FME Server was renamed to FME Flow. Since this article discusses features present in previous versions of FME, it will refer to both names interchangeably, using the appropriate product name based on the year the feature was introduced. For more information on the rebranding, see our website.

Content Overview

• Initial Troubleshooting
• Common Issues
  • “I’ve scheduled a task to clean up old files once a week, but the files are disappearing every day.”
  • “There are no tasks listed under System Configuration > System Cleanup > Tasks”
  • “My files are not being deleted even though I have a cleanup task scheduled.”
  • “The log file for Job ID <n> does not exist”
  • “Logs from FMEFlowJobSubmitter temporary engines are not being cleaned up”
• Are you still experiencing issues?
• Have ideas on how to improve this?

Initial Troubleshooting

• View the log file for Cleanup tasks in Resources > Logs > Service > current/old > fmecleanup.log and check for any errors
• If you have a distributed install, make sure all the FME Flow Components (Web Application/Core/Engines/Database) are running and have the correct permissions.

Common Issues

“I’ve scheduled a task to clean up old files once a week, but the files are disappearing every day.”

Your FME Flow host is likely low on disk space. FME Flow will monitor disk conditions and, if they fall below a set threshold, trigger aggressive cleanup tasks. You should look into freeing up some disk space on your FME Flow host machine.

Alternatively, you can increase the threshold at which an aggressive cleanup is triggered via the FME Flow Web UI under System Configuration > System Cleanup > Configuration, or disable low-disk conditions entirely by setting these values to zero. 

“There are no tasks listed under System Configuration > System Cleanup > Tasks”

As part of the installer, a series of batch files are provided, one of which creates your cleanup tasks. The installer should run these scripts automatically as part of the installation; however, if the cleanup tasks or anything else is missing from the FME Flow menu, you should manually run these scripts. 

1. Go to <FMEInstallDir>\Clients\utilities
2. Right-click on the AddCleanupTasks.bat (or .sh on Linux) and run as administrator
3. Restart FME Flow

If this script failed to run, it’s likely the others did as well; in this case, it is recommended you also run addPublishers, addSubscribers, addServices, and configureDemoWorkspaces before restarting FME Flow. 

“My files are not being deleted even though I have a cleanup task scheduled.”

View the task on the System Cleanup page of the Web UI and check the following: 

• The task is enabled
• If there is a filter on the task, do your files meet the criteria

On the Configuration page, you should also check the interval for cleanup tasks under normal conditions. By default, the cleanup tasks run only once a day. Therefore, even if you set a Tasks Maximum Age to (for example) 1 hour, this is the file age, but since the task runs only daily, you will still see files older than this. You may want to increase the Task Interval to remove files more frequently. 

“The log file for Job ID <n> does not exist”

There is a System Cleanup task that removes job logs; by default, it removes log files when they are 1 week old. If you try to view a log that is older than this cleanup date, the log file will have been deleted by this cleanup task, and this error will be displayed. If the job log you are trying to view was run in the last 7 days, your machine may have undergone an aggressive system cleanup task triggered under low disk conditions. 

Check out this article for more causes and resolutions. 

“Logs from FMEFlowJobSubmitter temporary engines are not being cleaned up”

When submitting jobs via the FMEFlowJobSubmitter transformer with the Submission Mode parameters set to Submit Jobs = In Sequence and Wait For Jobs To Complete = Yes, a child engine process is spun up to run those jobs. This engine uses the parent engine's name with the prefix @<ParentJobID>.   

For example, if the job containing the FMEFlowJobSubmitter was running on Engine 1 with ID 123, the child job would run on engine <HOST>_Engine1@123. When these engines start up, a log file is written to Resources > Logs > Engine > Current and is not handled by the system cleanup tasks. This can result in a large number of log files building up in this location. 

For both Windows and Linux

1. Run a Text Editor as administrator and open fmeEngineConfig.txt (located in <InstallDir>\Server)
2. Comment out, or remove the line starting with FME_SERVER_LOG_FILE

Are you still experiencing issues?

Please consider posting to the FME Community Q&A if you are still experiencing issues that are not addressed in this article. There are also different support channels available.

Have ideas on how to improve this?

You can add ideas or product suggestions to our Ideas Exchange.