Page tree
Skip to end of metadata
Go to start of metadata




System Control Menu

The System Control menu contains the following options, see image below.

System Control

With system control, 2 actions can be taken:

  • Index Files
  • Failover Status

Index Files

Troubleshoot option to trigger the index files for the ForecastingShells to be rebuilt when this has not happened automatically or in the unusual circumstance that the index files are corrupted. When the Reindex button is used a command is given to one of the available forecasting shells to rebuild the index files. The index files will be stored in the central database. Normally the index files are automatically expired after a day and will be automatically rebuilt.

Failover Status

With the "Set to Failover" button the current MC can be put into failover state.

Failover Priorities

The Failover priorities options can be used to configure on which Master Controller to run failover tasks in case of failover mode.

Failover mode can be activated by the following situations:

  1. when another MC is not reachable (systemmonitor has registered the remote queue proxy for the remote master-controller as being down).
  2. when an MC is marked to be failed over using manual failover.

Top master-controller

The Failover priorities page allows users to define in what situations a master-controller becomes a top master-controller. In order to become a top master-controller, the active status should be active, and all the higher ranked (i.e. smallest integer value) priority master-controllers should be unavailable or marked as manual failover. Once a master-controller has become the top master-controller it starts picking up failover tasks owned by other master-controllers.

Add/Edit Priority

The priorities can be created or edited (by clicking a row) and can only be altered to a unique number so that the order is changed.

Synchronization

Toggles the mc synchronization (pull) from another MC.

Common guideline on defining Failover priorities

As a rule, any Master Controller is in charge of running its own tasks. In addition, it takes over Failover tasks for which the owner is not visibly producing heartbeats indicating its health, and where no other healthy Master Controllers with better failover priority (lower value) are available. Each Master Controller has its own list of failover priorities, deciding which Master Controller becomes the next top MC in a Failover situation. For best behaviour, it is recommended to list all Master Controllers that are visible in the list of Failover Priorities. In order to prevent the local Master Controller becoming the top Master Controller for taking over its own "Suspend duty in Failover" Tasks in case of a manual failover,  it is common to specify all other Master Controllers first, and put the local Master Controllers' Failover Priority last.

Example:

MC

Failover priorities order by rank

Example
MC00MC01 MC02 MC00

The lowest failover priority integer value indicates the highest rank. For instance, when MC01=1 MC02=2 MC00=99 are defined on MC00 this indicates the failover priority order is "MC01 MC02 MC00'. In this case the TaskManager on MC00 runs "Suspend duty in Failover" Tasks when

  1. MC00 is not Failed or
  2. when MC01 and MC02 are Failed.

Three Master Controllers in a single synchronization pool with:

  1. MC00 and MC01 both own Failover Tasks.
  2. MC02 is not owning any Failover Tasks.
  3. MC00 and MC01 take over Failover Tasks from each other when the other is in Failover or down.
  4. MC02 has to take over Failover Tasks from MC00 and MC01 when MC00 and MC01 are down or failed over.

Each MC that is not in Failover mode can become top MC for Failover Tasks owned by other MCs as soon as no recent heartbeats (5 minutes threshold) are received from higher ordered remote MCs.



MC01MC00 MC02 MC01
MC02MC00 MC01 MC02


Status MC00Status MC01Status MC02Remarks
OKOKOKEach MC runs its own Tasks. No Failover Tasks are taken over.
FailedOKOKMC01 runs additionally all Failover Task from MC00. When in Manual Failover, MC00 no longer run its own "Suspend duty in Failover" Tasks but still runs all its other Tasks.
OKFailedOKMC00 runs additionally all Failover Task from MC01. When in Manual Failover, MC01 will no longer run its own "Suspend duty in Failover" Tasks but still runs all its other Tasks.
FailedFailedOKMC02 runs additionally all Failover Task from MC00 and MC01. When in Manual Failover, MC00 and MC01 will no longer run their own "Suspend duty in Failover" Tasks but still runs all the other Tasks they own.
OKOKFailedWhen MC02 is in Manual Failover, there is no visible effect. MC02 does not own Failover Tasks. MC02 is no longer allowed to take over Failover tasks from other MCs.
FailedOKFailedMC01 runs additionally all Failover Task from MC00. When in Manual Failover, MC00 no longer runs its own "Suspend duty in Failover" Tasks but still runs all its other Tasks. MC02 is no longer allowed to take over Failover tasks from other MCs.
OKFailedFailedMC00 runs additionally all Failover Task from MC01. When in Manual Failover, MC01 no longer runs its own "Suspend duty in Failover" Tasks but still runs all its other Tasks. MC02 is no longer allowed to take over Failover tasks from other MCs.
FailedFailedFailedWhen all three MCs are in Manual Failover, each MC runs its own Tasks. No Failover Tasks are taken over. MC00 and MC01 will run their own "Suspend duty in Failover" Tasks since they have become the top MC for their own Failover Tasks. The MCs are no longer allowed to take over Failover tasks from other MCs.
  • No labels