| scrollbar |
|---|
| Table of Contents |
|---|
System Control Menu
The System Control menu contains the following options, see image below.
System Control
With system control, 2 actions can be taken:
- Index Files
- Failover Status
Index Files
Troubleshoot option to trigger the index files for the ForecastingShells to be rebuilt when this has not happened automatically or in the unusual circumstance that the index files are corrupted. When the Reindex button is used a command is given to one of the available forecasting shells to rebuild the index files. The index files will be stored in the central database. Normally the index files are automatically expired after a day and will be automatically rebuilt.
Failover Status
Failover Priorities
The Failover priorities are defined locally on each Master Controller and also only processed by the local Master Controller. Failover priorities are only applicable when the owner of a Failover Task has the failover status.
Failover mode can be activated by the following situations:
- when another MC is not reachable (systemmonitor has registered the remote queue proxy for the remote master-controller as being down).
- when an MC is marked to be failed over using manual failover.
Top master-controller
The Failover priorities page allows users to define in what situations a master-controller becomes a top master-controller. In order to become a top master-controller, the status should be active, and all the higher ranked (i.e. smallest integer value) priority master-controllers should be unavailable or marked as manual failover. Once a master-controller has become the top master-controller it starts picking up failover tasks owned by other Master Controllers.
Add/Edit Priority
The priorities can be created or edited (by clicking a row) and can only be altered to a unique number so that the order is changed.
Synchronization
Toggles the mc synchronization (pull) from another MC.
Common guideline on defining Failover priorities
As a rule, any Master Controller is in charge of running its own tasks. In addition, it takes over Failover tasks for which the owner is not visibly producing heartbeats indicating its health, and where no other healthy Master Controllers with better failover priority (lower value) are available. Each Master Controller has its own list of failover priorities, deciding which Master Controller becomes the next top MC in a Failover situation. For best behaviour, it is recommended to list all Master Controllers that are visible in the list of Failover Priorities. In order to prevent the local Master Controller becoming the top Master Controller for taking over its own "Suspend duty in Failover" Tasks in case of a manual failover, it is common to specify all other Master Controllers first, and put the local Master Controllers' Failover Priority last. If there is a tertiary backup system (without Failover tasks of itself) that needs to take over from MC00 and MC01, then the tertiary system is always specified last.
Example:
...
Failover priorities order by rank
...
The lowest failover priority integer value indicates the highest rank. For instance, when MC01=1 MC00=2 MC02=99 are defined on MC00 this indicates the failover priority order is "MC01 MC00 MC02'. In this case the TaskManager on MC00 runs "Suspend duty in Failover" Tasks when
- MC00 is not Failed or
- when MC01 and MC02 are Failed.
Three Master Controllers in a single synchronization pool with:
- MC00 and MC01 both own Failover Tasks.
- MC02 is not owning any Failover Tasks.
- MC00 and MC01 take over Failover Tasks from each other when the other is in Failover or down.
- MC02 has to take over Failover Tasks from MC00 and MC01 when MC00 and MC01 are down or failed over.
Each MC that is not in Failover mode can become top MC for Failover Tasks owned by other MCs as soon as no recent heartbeats (5 minutes threshold) are received from higher ordered remote MCs.
...