FME Flow Deployment Life Cycle for the Enterprise

Introduction

For this article, the deployment life cycle refers to the framework that describes the planning and maintenance involved in implementing FME Flow (formerly FME Server) software within your organization. This article introduces the deployment life cycle of an enterprise FME Flow, from Development to Production, and any system in between.

Life Cycle Stages

Deploying FME Flow in the enterprise may involve several systems before the application goes live in a production environment. Typically, FME Flow begins in a Development (DEV) environment, migrates to a Testing (TEST) environment, and then proceeds to a live, or Production (PROD) environment for everyday use in the enterprise. Other environments, such as STAGE and PREPROD, are also typical.

The chosen FME Flow architecture model and configuration should be replicated across each environment in the life cycle, and the essential contents should be migrated correctly to ensure consistent operation of the FME workflows.

Life Cycle Deployment & Migration

When migrating an FME Flow through the enterprise life cycle, the efforts undertaken in the DEV environment must carry through to the next system. These efforts may include new workspaces or updates to existing ones, and changes to user management, security, automations/notifications, schedules, resources, connections, projects, streams, server apps, engine & queues, and other third-party integrations with FME Flow. These changes may have been made via the FME Flow Web User Interface or involve one of FME Flow’s configuration or properties files. To account for these changes, each scenario requires somewhat different considerations, or workflows, for implementing a migration.

Replicating FME Flow from one environment to the next may also involve a regularly scheduled software upgrade process or fewer software upgrades that only occur when new features and enhancements are added to the FME Flow Application. As an enterprise, you must determine the breadth of the migration that suits your business requirements.

Migration Considerations

• Do you have the required software installers?
  • FME Form, FME Flow, and any other required 3rd-party software? Have you recorded or acquired the FME Flow Licensing information (serial number) so the new software can be licensed after a software upgrade? Does the system where FME Flow is being installed have access to the internet (Automatic Licensing URLs) to permit automatic licensing, or will manual licensing be required? Licensing Options
• Have you read the latest online documentation for software installer options?
  • Installation Options
• Are you using an external web application server as the FME Flow Web Application?
  • This does not apply to an Express install, and is optional for a distributed/fault-tolerant install
  • Provide a Web Application Server
• Are you using an external database like PostgreSQL, SQL Server, or Oracle? 
  • This does not apply to an Express install and is optional for a distributed install
  • FME Flow Database
• If providing your own database, have the optional Database Client Software and FME Flow database scripts been identified?  
  • Have the database login credentials been obtained?
    • The required database scripts to create the FME Flow Database are accessible after installing FME Flow.  
    • Provide a Database Server
• Will you be using domain service accounts for FME Flow services?
  • Distributed environments often require the use of domain service accounts.
  • If upgrading an existing FME Flow environment, note the services 'log on as' settings.
  • Directory and Account Permissions
• Have any changes been made to the configuration files of FME Flow or the properties file of the Web Application Server?
  • For example, has FME Flow been configured for HTTPS?
  • Configuring for HTTPS
• Are you required to maintain the Job logs for historical records?
• For existing systems
  • Backup job history (if necessary).
  • Backup job log files (if necessary).
  • Record the 3rd party software installed that FME Flow uses.
  • Record OS system environment variables; for example, FME_TEMP
  • Record OS system PATH entries, if applicable; for example, Oracle Instant Client.
• Contact your local reseller or Safe Software Support for any questions or clarifications.

Migration Limitations

The following are key configuration considerations when migrating FME Flow and the limitations for each.

Configurations Stored in the FME Flow Database

There is no way to determine the difference between any two FME Flow Configurations automatically. Typically, the configuration files should not be changed, but there are instances when manual changes are made, such as HTTPS configuration. In this case, it is necessary to identify the differences and reinforce why tracking changes to the environments is recommended.

Some configurations, including processMonitorConfigEngines.txt and processMonitorConfigCore.txt, are stored in the FME Flow Database. These configurations are migrated with a default backup and restore operation. If, however, the NODE_OVERWRITE directive is set to true in the configuration file(s), the config file is read during each startup of FME Flow and not stored in the database.  

Ensure you are aware of the changes made to the config file and apply those changes to the new environment, but only when necessary, as new software versions may not require the change.  If in doubt, confirm with Safe Software Support or your FME Reseller.

FME Flow Configuration Files ​​​​​​​
Much information about an FME Flow installation remains in configuration files, which are not transferred during a migration. Changes to these files must be tracked and made to other systems in your life cycle. Consider backing up copies and using versioning software to track changes. DO NOT replace config files from older versions of FME Flow, as newer config files may have additional settings that do not exist in the older versions.  We continue to improve the product and move the configuration to the FME Flow Database.  It is important to ensure the file configuration changes are applicable to the new software.

Web Applet Properties Files
These files do not migrate automatically, and any changes must be recorded. Consider backing up copies and using versioning software to track changes. Please don't replace the properties files from older versions of FME Flow. Often, these files are not modified and can be ignored.  

HTTPS/SSL and Single Sign-On
If you have made SSL configuration changes, ensure you have the necessary information to apply the settings to the new software.  We do not support copying files to the new software or system.  We recommend repeating the steps in the article Configuring for HTTPS in the new software for each system. Likewise, SSO configuration must be performed again if restoring to a new system. If FME Flow is installed on the same system, you must reapply changes to the Tomcat Properties File for SSO.

Security
User Configuration, Users, Roles, Tokens, Azure AD, SAML, and Active Directory settings are backed up. The new system will not affect the new ‘admin’ superuser; for example, the password will not be reset to the old system’s ‘admin’ password. After migration, confirm that the registered apps or IDP use the correct hostnames (this applies to Azure AD and SAML).  

Database and Web Connections for Workspaces
Database Connections for workspaces are migratable; however, be aware of any database aliases used in the connection syntax - for example, Oracle’s tnsnames entries or database instance names that may not exist in the next environment and may present problems. Using EZConnect syntax for Oracle databases avoids using tnsnames and may be the preferred approach.

Different environments may also affect other database types (SQL Server, PostgreSQL). Understanding how the Engine Service “Log On As” user was set is important and ensuring the new system is configured similarly, especially if your connections use Windows Authentication. Web Connections will be migrated as well. There are instances where you’ll need to revalidate them after migration.

External Resources
Often, supporting data sources or other template files are used in FME workspaces. These resources are not migrated if they are not published to FME Flow. If these files are not in a network location, for example, accessed via a UNC path, you must manually copy or provide other automated provisions to move these files. (An external FME workspace can use FileCopy to move files if network connectivity exists between environments).

FME Flow Resources
Depending on how users have published data to the FME Flow, there can be very large files/folders within the FME Flow Data Resource Folder or Workspace Repositories.  This can impact the file size of the backup file, and in some cases, cause the backup or the restore to fail.  If the backup results in a very large .fsconfig file (> 4GB), you may want to investigate if there are files within the Data Resource folder or in the Repository Folder that can be temporarily removed before doing the backup and restore process.

Additionally, if Resources consist of AWS S3 Buckets or UNC Paths, only the configuration is backed up, not the files, as these are outside the storage of FME Flow.

Please contact Safe Software Support if you have problems with the backup or restore processes, and you have a very large .fsconfig file.

Pre-requisites: Admin Guide reading:

• Reviewed Before you Begin section.
• Reviewed Express Install for Windows or Linux section.
• Reviewed Upgrading FME Flow section.
• Reviewed Workspace Life Cycle for the Enterprise.