Job Routing for Maintenance Cycles on a Fault-tolerant Installation

Introduction

This article provides instructions for setting up job routing when performing maintenance on Fault-tolerant installations. This is especially helpful when you have to take a host machine offline for scheduled maintenance and want to ensure that jobs will be rerouted to the remaining active core.

For the purpose of this demo, it is assumed that this is a Fault-Tolerant installation where the Flow engines reside on the same machine as the Cores. Maintenance will need to be scheduled at different times of the day, keeping at least one core online at all times. 

You would like to configure FME Flow to ensure that jobs will be routed to the Flow engines on the core host that is available during maintenance. This will be achieved in this tutorial by - 

• Creating new Queues, one for each maintenance period. 
• Configuring Engine Assignment Rules, one for each queue, to allocate engines to the new queues. Engines can be allocated in two ways: 
  • Assign queues based on engine names. This works well in configurations where engine names do not change (in local or cloud-based installations), but in Docker- or Kubernetes-based installations, engine names change whenever engine containers change. 
  • Assign queues based on engine properties (e.g., engine hostname, OS, engine type). 
• Creating Active Periods for each maintenance period and assigning them to their respective Engine Assignment Rules.

In this example, the engine will be assigned using the property type ”Name”. The Engine Assignment Rules will be used to remove engines from the Default queue during Active Periods, leaving only available engines in the Default queue. This will prevent any jobs from being submitted to offline engines during maintenance and allow users to continue submitting jobs to the Default queue without manual intervention.

Note: If no engines are explicitly set to be used by a queue in FME Flow, all engines are routed to the Default queue with priority set to 5 (Medium). 

Step-by-Step Instructions

Part 1: Configure Queues

1. Navigate to “Engine Management” under the “Admin” section and select “Queues” on the FME Flow Web UI. 

   

2. Click on “Create ” to create a new queue. 

   

3. Fill in the appropriate queue name, description, and priority, and click OK. For our demo, an “Afternoon Maintenance” queue with a priority of 7 (Medium) will be created. Basically, any priority higher than the “Default queue” that has a priority of 6 (Medium) should work. We can preview the priorities of other existing queues in the system by clicking “Preview queue priorities”.

   

4. Repeat this and create a new "Morning Maintenance queue. You should now have three queues. Afternoon Maintenance, Morning Maintenance, and Default.

   

Part 2: Configure Engine Assignment Rules

1. Navigate to “Engine Management” under the “Admin” section and select  “Engine Assignment Rules” on the FME Flow  Web UI and click "Create"

   

2. Configure all the parameters as follows, and then select OK: 

• Name: Afternoon Maintenance
• Type: Name
• Rule: Select “Standard” from the drop-down menu 
• Engines: Select all engines from one of the cores. E.g. Core 1 (This will be the core that's scheduled to go offline during this period.)
• Queue: Afternoon Maintenance (e.g. <QueueCreatedInPart1>) 
• Enabled: On

• Active Periods: Once an active period is created, you can select the option in the drop-down menu. We will come back to this later. You can leave this blank for now.

  

3. Repeat this for the Morning Maintenance Queue. Assign the engines from the second core machine to this rule.

Part 3: Configure Active Periods

1. Navigate to “Engine Management” under the “Admin” section and select  “Active Periods” on the FME Flow  Web UI and click "Create".

   

2. Fill in the appropriate queue name, description, Start Date, Start Time, End Time, and Recurrence.

   

3. Under Engine Assignment Rule, choose the corresponding Engine Assignment Rule you created in Part 2. E.g. Afternoon Maintenance

   

4. Repeat this for the Morning Maintenance Engine Assignment Rule.

   

Part 4: Verify the Configuration

1. Once the Queue, Engine Assignment Rules, and Active Periods have been set, navigate to the “Overview” or “Queues” page to check the configuration. Outside of the Active Periods, you should see all engines available in the Default Queue.

   

2. When an Active Period kicks in, you will see the engines on the core host that will go offline for maintenance (e.g., Core 1) be assigned to the Maintenance Queue and removed from the Default Queue. The Default Queue will now only be assigned engines from the available core. Any job that you run that isn't explicitly routed to a different queue will continue to run as normal using only the available engines. 

   

3. Once the Active Period is over after maintenance is complete and the core is back online, you will see the engines from that core reassigned to the Default queue and available to run jobs.