Search

Leverage Projects within CI/CD process flow

Merline George
Merline George
  • Updated

FME Version

Introduction

This article delves into the latest 2023 enhancements within FME Flow Projects, highlighting how to leverage projects to create CI/CD workflows. Imagine a scenario where your CI/CD pipeline flows through distinct Development, Testing, and Production FME deployments. In such a setup, the goal is to facilitate the smooth migration of Projects between environments, reducing the need for manual intervention.

Furthermore, these improvements benefit organizations utilizing FME Flow that have yet to adopt CI/CD methodologies. They offer visibility and transparency throughout the process of manually working with Projects.
 

New 2023.0 Features 

The new Project features in FME 2023.0 are designed to facilitate efforts in Continuous Deployment. Some of the features are - 

  • Full Nested Dependency support ensures that when adding an FME item during project creation, FME automatically identifies, displays and includes the FME item’s dependencies like web and database connection, packages, deployment parameters etc, into the project; thereby automating the process. 

 

1.png

 

Previously, adding an item into a Project would not automatically add all the dependencies and accidentally missing some of these dependencies could cause runtime issues in the other environments post migration. This would also have been a roadblock to a smooth CI/CD workflow. 
 
  • Deployment Parameters, another component of the CI/CD workflow, can also be added to a project explicitly. 

 

2.png
 
  • Any changes made on the items in the Project post-creation [like adding/changing/deleting workspaces in an automation in the project, adding dependency to a web connection for a workspace in the project] will automatically be updated in the Project.  

 

  • The Import Preview page provides visibility and transparency into what changes the import will make to the target Flow instance. We can now see what FME items are a part of a Project, its type and its owner. We can also see if these will be new items on the target Flow instance or if they already exist and will therefore be overwritten by this project import action. 
If the items in the project already exist on the system, we can also choose to allow the overwrite or not. 
 
3.png

Usually, certain items within the project are specific to the environment [such as any connections, deployment parameters, tokens, or users]. In that case, you can choose to import or exclude them and instead create those items specific to the target Flow instance’s environment. 
 
4.png

Although the primary objective of CI/CD is to reduce manual inspections and procedures, it serves as an excellent tool for initially assessing the extent of the import action.
 
  • FME Flow CLI can be utilized directly for facilitating the migration of projects across different environments. 
Initially, you'll need to manually publish the workspace from your FME Form to FME Flow within your working environment. However, after this initial step, the migration process across environments can be automated. You can read more about how to achieve this in the article -Getting Started With DevOps

  

Automating Notifications for project import and export 

We can integrate CI/CD process with automated alerts to inform stakeholders or team members of the results of tests, validations or any issues that arise during Project import/export. 

  • Using System Events notification

We would first set up System Email to send out notifications from FME Flow instance under Admin>System Configuration>Network & Email>System Email.  

5.png

We can now enable Email notifications for “Project Exported” and “Project Imported” System Events under Admin>System Configuration>System Events. This should now trigger an email whenever a project is exported from or imported into the system.
 
6.png
7.png

Please note that the email notification is triggered regardless of whether the import/export job was successful or it failed. 
 

  • Using Automation

In addition to the above configuration, if you only want to setup notification for specific scenarios such as –
  • For certain export/import jobs, 
  • Only if the job was successful or failed 
  • Only if the action was project import or project export 
– we can use Flow Automation to filter out the Project attributes such as Project Name [<ProjectName>], Status [SUCCESS/FAILURE], Event.title [Project Exported/Project Imported] etc and trigger emails as required. 
 
A simple example would be as shown below, with an FME Flow System Event (received) trigger, Filter for Event Title “Project Exported” and external action Email (send)

8.png
9.png

10.png

 

Testing a project

Once a project has been imported successfully, it's recommended to execute a test run of the imported FME objects and perform data quality assessments on your associated databases and repositories.  You can however choose to automate running a workspace that is within a project through the command line interface (CLI) by utilizing the “fmeflow run” command. This won't give you the full assurance of data quality but will confirm a workspace held in a project ran successfully.
 

Updating a project

During import, if a project with the same name already exists (albeit another version), the existing project and the items within the project will be overwritten [you can choose to 
allow or exclude items to be overwritten if they already exist]. No items will be removed from the FME Flow instance, however items may be removed from the existing project in case the newer version does not contain the same items.

11.png

 

Removing Projects / Rolling back


In some cases, you may need to revert an import after deploying a newer version of the project. It might become necessary to restore the previous version of the project, potentially due to a test that did not succeed.
There are different options that you can select while removing a project- Removing the project only, removing the project with project items or removing the project with project items and dependencies. Please note that if we remove an updated project and you choose to remove all project items from FME Flow,  it will not revert back to the original project or item version and they will be deleted from the instance. Alternatively, If you choose to only remove the updated project from FME Flow but keep the project items that were a part of it, it will delete the project from the instance but keep the current version of the items.
Hence, it is advisable to maintain backups of the project's previous versions, so you have the option to remove the current project and then restore the previous version of the project to return it to its original state.
 12.png
 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.