| scrollbar |
|---|
| Table of Contents |
|---|
System Control Menu
The System Control menu contains the following options, see image below.
System Control
With system control, 2 actions can be taken:
- Index Files
- Failover Status
Index Files
Troubleshoot option to trigger the index files for the ForecastingShells to be rebuilt when this has not happened automatically or in the unusual circumstance that the index files are corrupted. When the Reindex button is used a command is given to one of the available forecasting shells to rebuild the index files. The index files will be stored in the central database. Normally the index files are automatically expired after a day and will be automatically rebuilt.
Failover Status
Failover Priorities
The Failover priorities options can be used to configure on which Master Controller to run failover tasks in case of failover mode.
Failover mode can be activated by the following situations:
- when another MC is not reachable (systemmonitor has registered the remote queue proxy for the remote master-controller as being down).
- when an MC is marked to be failed over using manual failover.
Top master-controller
The Failover priorities page allows users to define in what situations a master-controller becomes a top master-controller. In order to become a top master-controller, the active status should be active, and all the higher ranked (i.e. smallest integer value) priority master-controllers should be unavailable or marked as manual failover. Once a master-controller has become the top master-controller it starts picking up failover tasks owned by other master-controllers.
Add/Edit Priority
The priorities can be created or edited (by clicking a row) and can only be altered to a unique number so that the order is changed.
Synchronization
Toggles the mc synchronization (pull) from another MC.
Common guideline on defining Failover priorities
As a rule, any Master Controller is in charge of running its own tasks. In addition, it takes over Failover tasks for which the owner is not visibly producing heartbeats indicating its health, and where no other healthy Master Controllers with better failover priority (lower value) are available. Each Master Controller has its own list of failover priorities, deciding which Master Controller becomes the next top MC in a Failover situation. For best behaviour, it is recommended to list all Master Controllers that are visible in the list of Failover Priorities. In order to prevent the local Master Controller becoming the top Master Controller for taking over its own "Suspend duty in Failover" Tasks in case of a manual failover, it is common to specify all other Master Controllers first, and put the local Master Controllers' Failover Priority last.
Example:
| MC | Failover priorities order by rank | Example |
|---|---|---|
| MC00 | MC01 MC02 MC00 | The lowest failover priority integer value indicates the highest rank. For instance, when MC01=1 MC02=2 MC00=99 are defined on MC00 this indicates the failover priority order is "MC01 MC02 MC00'. In this case the TaskManager on MC00 runs "Suspend duty in Failover" Tasks when
Three Master Controllers in a single synchronization pool with:
Each MC that is not in Failover mode can become top MC for Failover Tasks owned by other MCs as soon as no recent heartbeats (5 minutes threshold) are received from higher ordered remote MCs. |
| MC01 | MC00 MC02 MC01 | |
| MC02 | MC00 MC01 MC02 |
| Status MC00 | Status MC01 | Status MC02 | Remarks |
|---|---|---|---|
| OK | OK | OK | Each MC runs its own Tasks. No Failover Tasks are taken over. |
| Failed | OK | OK | MC01 runs additionally all Failover Task from MC00. When in Manual Failover, MC00 no longer run its own "Suspend duty in Failover" Tasks but still runs all its other Tasks. |
| OK | Failed | OK | MC00 runs additionally all Failover Task from MC01. When in Manual Failover, MC01 will no longer run its own "Suspend duty in Failover" Tasks but still runs all its other Tasks. |
| Failed | Failed | OK | MC02 runs additionally all Failover Task from MC00 and MC01. When in Manual Failover, MC00 and MC01 will no longer run their own "Suspend duty in Failover" Tasks but still runs all the other Tasks they own. |
| OK | OK | Failed | When MC02 is in Manual Failover, there is no visible effect. MC02 does not own Failover Tasks. MC02 is no longer allowed to take over Failover tasks from other MCs. |
| Failed | OK | Failed | MC01 runs additionally all Failover Task from MC00. When in Manual Failover, MC00 no longer runs its own "Suspend duty in Failover" Tasks but still runs all its other Tasks. MC02 is no longer allowed to take over Failover tasks from other MCs. |
| OK | Failed | Failed | MC00 runs additionally all Failover Task from MC01. When in Manual Failover, MC01 no longer runs its own "Suspend duty in Failover" Tasks but still runs all its other Tasks. MC02 is no longer allowed to take over Failover tasks from other MCs. |
| Failed | Failed | Failed | When all three MCs are in Manual Failover, each MC runs its own Tasks. No Failover Tasks are taken over. MC00 and MC01 will run their own "Suspend duty in Failover" Tasks since they have become the top MC for their own Failover Tasks. The MCs are no longer allowed to take over Failover tasks from other MCs. |