Delft-FEWS Admin Interface - System Control

System Control Menu

The System Control menu contains the following options, see image below.

System Control

With system control, 2 actions can be taken:

• Index Files
• Failover Status

Index Files

Troubleshoot option to trigger the index files for the ForecastingShells to be rebuilt when this has not happened automatically or in the unusual circumstance that the index files are corrupted. When the Reindex button is used a command is given to one of the available forecasting shells to rebuild the index files. The index files will be stored in the central database. Normally the index files are automatically expired after a day and will be automatically rebuilt.

Failover Status

Failover Priorities

The Failover priorities options can be used to configure on which Master Controller to run failover tasks in case of failover mode.

Failover mode can be activated by the following situations:

1. when another MC is not reachable (systemmonitor has registered the remote queue proxy for the remote master-controller as being down).
2. when an MC is marked to be failed over using manual failover.

Top master-controller

The Failover priorities page allows users to define in what situations a master-controller becomes a top master-controller. In order to become a top master-controller, the active status should be active, and all the higher ranked (i.e. smallest integer value) priority master-controllers should be unavailable or marked as manual failover. Once a master-controller has become the top master-controller it starts picking up failover tasks owned by other master-controllers.

Add/Edit Priority

The priorities can be created or edited (by clicking a row) and can only be altered to a unique number so that the order is changed.

Synchronization

Toggles the mc synchronization (pull) from another MC.

Common guideline on defining Failover priorities

As a rule, any Master Controller is in charge of running its own tasks. In addition, it takes over Failover tasks for which the owner is not visibly producing heartbeats indicating its health, and where no other healthy Master Controllers with better failover priority (lower value) are available. Each Master Controller has its own list of failover priorities, deciding which Master Controller becomes the next top MC in a Failover situation. For best behaviour, it is recommended to list all Master Controllers that are visible in the list of Failover Priorities. In order to prevent the local Master Controller becoming the top Master Controller for taking over its own "Suspend duty in Failover" Tasks in case of a manual failover,  it is common to specify all other Master Controllers first, and put the local Master Controllers' Failover Priority last.

Example: Three Master Controllers in a single synchronization pool with MC00 and MC01 owning failover tasks, MC02 has to take over when MC00 or MC01 are gone but MC02 is not owning any Failover Tasks.