FME Flow Administration: Backup, Migrations and Upgrades

Introduction

Performing backups and migrations is a common task for an FME Flow system administrator. A backup is carried out using the tools on the “Backup & Restore” page of the FME Flow web interface. A migration is achieved by performing a backup and then restoring the backup on a different system or the same system post a version upgrade.
 

Backing up and Restoring FME Flow

Backup and Restore process is an important part of FME Flow upgrades. By backing up your current server and restoring the configurations and settings to the new version, you can keep your test and production environments in sync and allow a quick restoration after an upgrade. 

FME Flow also provides a workspace that you can configure to perform regular (scheduled) backups of your server configuration to an FME Flow configuration (.fsconfig) file. You can choose to schedule a backup of your server configuration for the time range that works best for you and your organization. The time ranges from every second to every year. You can view the backup and restore operations history under the “History” tab of the “Backup & Restore” section. 

It is highly recommended that you verify the server backup file first by installing a new instance of FME Flow before you uninstall the current FME Flow. Once you uninstall, you cannot restore the setup unless you regularly perform backups of your entire file system. If you're doing an upgrade on the same machine, you won't be able to test this as you cannot install multiple versions of FME Flow on the same machine. However, in such cases, we recommend you validate the backup file by -

• Performing a restore on the current system using a recent .fsconfig backup file 
• Navigating to the History” tab of the “Backup & Restore” section and confirming a successful configuration restore

Advanced Tips and Tricks 

• If encryption is enabled on the backed-up FME Flow, it must also be enabled on the restored FME Flow, and it should use the same custom encryption key. Please ensure you have your custom encryption key downloaded and safely saved.  As of FME Flow 2023.1, encryption is enabled by default, so it is important that you download the encryption key before creating any backups to ensure you can restore the backup on a new installation.

• Resource folders generally contain large amounts of data that require considerable time to back up, and this results in large configuration files. It is recommended that you define large folders as non-migratable. Resources whose definitions and files are stored with FME Flow and configured for migration. The DATA, ENGINE, and DASHBOARD Resources are configured for migration by default. To determine if your resource folders are migratable, open the fmeFlowConfig.txt (formerly fmeServerConfig.txt) configuration file. Locate your resources under Shared Resource Management. Resources with a SHAREDRESOURCE_ISMIGRATABLE_* value of true are migrated. Following backup and restore, manually migrate these folders to the new FME Flow. Alternatively, you can host big data on a remote file share and point the new FME Flow to that path via UNC path.

• Any messaging to or from Automations triggers or external actions or Notification Service topics, is disabled during the restore operation. This prevents the restore operation from triggering unintended notifications from any actively-configured triggers or publications.

Which FME Flow Configuration Components in a Backup?

Please see the full list of FME Flow components that are recorded in a backup and can be migrated to a new Flow installation.

The following components of your FME Flow are not migrated:

• Core and Job Logs: Core and Job logs cannot be migrated and must be manually backed up. If you want to backup the entire logs folder, you can find it on your file system inside the resources folder of the FME Flow System Share [express install default location is C:\ProgramData\Safe Software\FME Flow\resources\logs].

• Job ID: As with Job Logs, the Job ID is not migrated between Flow instances. Job ID will start again from 0 on a new install.

• Version Control: If the FME Flow you are backing up has Version Control enabled, the versioning information will not be migrated. If Versioning is important, ensure to keep this information in a Remote Repository like GitHub.If Version Control was enabled on the backed-up configuration, and configured with a remote Git repository, the restored server does not automatically synchronize with the repository. The restored server does not automatically perform a “Fetch From Remote” operation. To avoid errors, be sure to fetch updates from the repository before pushing updates to the repository.

• Configuration Files: Any changes to FME Flow that are made through configuration files, like HTTPS/SSL, will not be migrated.

Backup Configuration Files

Before proceeding with the primary task of backing up your FME Flow through the web interface, we recommend manually copying over FME Flow Configuration Files to a location outside of your FME Flow installation directory to be used as reference, if required. The files to copy over will also depend on the level of customization of your FME Flow installation. These files contain configurations for the FME Flow Database, HTTPS and other miscellaneous settings for the Transformation and Repository Managers. These configurations are not included in the primary backup procedure. 

To restore any changes/customizations that were made in these config files, examine the old and new configuration files together and update configurations in the new file as you come across them and use line commenting for tracking specific changes. 

Please note that when restoring FME Flow configuration files that are manually backed up, do not replace or overwrite the files in the new installation and go through each configuration file instead. The FME Flow directory structure, configuration files structure and parameters may change between releases. 

In addition to this list of configuration files, please backup the following files - 

• <FMEFlowDir>\Server\config\<subscribers_or_publishers>\websocket.properties used to configure the FME Flow WebSocket server.
• <FMEFlowDir>\Utilities\pgsql\data\pg_hba.conf used to open the database connections for express install.
• <FMEFlowDir>\Utilities\tomcat\conf\server.xml used to configure web application server port, and connector protocols.
• <FMEFlowDir>\Utilities\tomcat\conf\web.xml used to configure international encodings.
• <FMEFlowDir>\Utilities\tomcat\conf\context.xml used to configure FME Flow web application.

Upgrading FME Flow

Full FME Flow Upgrade

There are multiple approaches to upgrading your complete FME Flow installation, each with its own advantages and disadvantages. Please read the documentation to decide which scenario listed below is right for you.

• Upgrading to a second machine with a different hostname
• Upgrading to a second machine using the same hostname
• Upgrading FME Flow in-place, on the same machine

FME Engine Only Upgrade

FME Engines are the component of FME Flow that perform the actual data translations and transformations. There might be instances you might want to upgrade only your FME Engines and not any other FME Flow components.  Please see the documentation for more information about FME Engine Only Upgrade. 
 

FME Flow Projects

You can use the Projects page to bring together FME Flow items into groups that share a common solution, or 'project,' in your organization. By grouping items in projects, you can reference them as one easy object. A project may then be exported and subsequently imported to either the same, or another instance of FME Flow.

The primary benefit of projects is the ability to migrate them between FME Flow servers, especially when deploying an FME Flow scenario through the enterprise life cycle, from development to testing/staging to production. Instead of backing up and restoring an entire FME Flow configuration, you can simply create and export the project and import it to another FME Flow. The target server configuration remains intact, with the addition of the new project and all of its contents.
 

What Can a Project Hold?

A project can include most FME Flow objects you create or upload. Please see the full list of FME Flow objects.
 

Creating a Project

You can create a project under the “Projects” > ”Create Project” section.  Please see the detailed steps to create a project. When the user account creating a project –

• Has the fmesuperuser role- 
  • They will be able to add any FME object on the server, including any workspace published, any automation created, any user and any user's API token to a project. 
  • To add users explicitly to a project, you must belong to the fmesuperuser role.

• Does not have a fmesuperuser role but they published/created/owned the FME object or the FME object has been shared with them - 
  • They will be able to add any FME object on the server that has been published/created/owned by or shared with the user. 
  • They will not be able to add any user at all. If you are not a superuser, only your own user account can be added ( implicitly) to the project file. 
  • They will only be able to add their own API token. Please note that users with “fmeadmin” role will be able to view/add other user’s token to a project [users with “fmeauthor” role will not be able to do this]. However, they will get the error “Exporting user token for user account requires the fmesuperuser role” if they try to save the project with another user’s API token. At this point, you can choose to “Save Anyway”. But this user creating the project will not be able to export it and will see the error message “Selected project cannot be exported until the reported issues are resolved.”

Adding Resource folders includes the folder and all of its contents. API tokens owned by users are not included in projects to which they are added, unless the tokens are explicitly added.

If the project contains any automations, schedules, or server apps, FME Flow checks if any items required for these components to run, such as workspaces, are also included in the project. 

If not included, the “Add Dependencies” dialog appears. To review, expand the drop-down beside the component with missing dependencies. By default, any dependencies will be added to the project unless you clear their respective checkboxes. To proceed with adding dependencies to the project as marked, click “Add Dependencies”. To proceed without adding any dependencies to the project, click “Skip”.
 

Advanced Tips and Tricks

The dependency check that FME Flow performs on projects is not comprehensive. FME Flow checks for the following dependencies:

• Automations: topics, workspaces, resources, resource connections. 
• Must be included manually: database connections, web connections, packages 
• Schedules: topics, workspaces
• Flow apps: workspaces, database connections, web connections, resource connections, tokens, users that own the associated tokens

Ensure that any other dependencies are included in your project and, if necessary, add them later.
 

Exporting and Importing a Project

You can export the contents of FME Flow Projects and import the projects to the same instance or another instance. Please note that the FME Flow you are importing the project to must be of the same or newer build than the FME Flow exported from. You can view project import/export under “History” tab under “Projects” section. For more information, see Exporting and Importing Projects.

When the user account exporting and importing a project –

• Has the fmesuperuser role-
  • They will be able to export and import any FME object on the server, including any workspace published, any automation created, any user and any user's API token to a project.
  • Users added to the project are created in the new FME Flow if they do not exist. The associated tokens [if they were added to the project] are also created. The same permissions the user has on the current system apply to the system to which the project is imported, but only on items in the project. Any other permissions the user has on the current system do not carry over. 
  • User importing the project becomes the owner of any items in the project, if the project does not include the user accounts of the original owners.

• Does not have the fmesuperuser role and the user exporting and importing the project are both different since the user importing the project into the new instance does not exist in the old server instance-
  • The project import will fail with the error message “Unable to import project with user due to the following issues: Does not have permission” to add/update user account, user role, API token, or to publish to the repository, which requires the fmesuperuser role. The user importing the project is expected to be the owner of all the FME objects being imported.  

• Does not have the fmesuperuser role, and the user exporting and importing the project are the same i.e the user importing the project into the new instance also exists in the old server instance- 
  • The project import will be successful if the user importing the project is the owner of all the FME objects being imported. 

If a project contains Flow Apps without their associated API tokens, those apps are disabled on the target system until explicitly enabled.

The user who performs the import becomes the owner of any items in the project, if the project does not include the user accounts of the original owners or if those user accounts do not exist on the target server. If the project contains any items of the same name on the system to which the project is imported, and the user performing the import does not own those items, the import fails.

You may want to update the project version before you export it. The specified version is appended to the .fsproject file that is exported.
Importing a project triggers only the “Project Import” system event, if enabled. But if a new repository is created on the FME Flow as a result of importing a project, the “Repository Create” event is not triggered.
 

Migrating Projects for Other Users (fmesuperuser only)

When you include users in a project, you can backup and restore a project for those users to work with on another system. Any permissions a user has on the current system are granted on the system to which the project is restored if both of the following conditions are true:

• The permission is on an item in the project.
• The permission was granted to the user directly rather than through membership in a role.

Any other permissions a user has on the current system do not carry over.
 

Viewing and Editing Projects

To view or edit a project, click on it to open the Edit page. Alternatively, select the project and click Edit. You can:

• Edit the project name and description.
• Add and remove items.
• Get information about items
  • Select an item under Contents and click the information icon to the far right for item details.

Sharing Projects

To share a project, click the Share with Others icon. Through sharing, you can grant levels of permissions on an item to other users. You can share an item if you own it, or if you are a user with Manage permission in Security, such as an administrator. For more information about sharing and ownership, see Role-Based and User-Based Access Control.
 

Removing Projects

If a project changes after it is exported, but before it is imported back to the same system, you may want to revert it to its original state so that it matches the exported version of the project. For more information, see Removing Projects.
 

Projects and FME Hub

FME Flow Projects are now available on FME Hub.
You can download a project from FME Hub and import it to your FME Flow, or export and upload a project to the FME Hub that you wish to share.

Upload a project to FME Hub using the correct FME Hub details listed below. This option can be accessed by expanding the FME Hub (Optional) dialog box when editing a project.

• FME Hub Publisher ID: When you publish to FME Hub, you will do so under a publisher. Enter the publisher value here.
• FME Hub Project ID: This is a unique identifier for the project.
• Project Icon: This icon will be used on the FME Hub when the project is listed.

What's Next? 

After FME Flow has been configured and backups have been planned, you can start adding users to your FME Flow. For further guidance, see FME Flow Administration: User Administration.