08 System Monitor

System Monitor

The System Monitor display allows for viewing the status of the system. The behaviour of the System Monitor is different depending on the use of the system, i.e. whether you are using the system in Stand Alone mode or as an Operator Client.

In the Standalone mode the System Monitor will display the Log Browser, Forecaster Notes, Batch Forecast tab, Import status and since 2021.02 also Running forecasts. On overview of the tabs available in the System Monitor of a Standalone system is illustrated below.

In an Operator Client the System Monitor will show a number of tabs for monitoring various aspects of the live system. On overview of the tabs available in an Operator Client when logged into a live system is illustrated below.

Each of the System Monitor tabs is detailed in the following paragraphs.

Log Browser

The Log Browser contains a spreadsheet like form with a list of system messages. Messages can be sorted based on user defined criteria. Fatal and severe system errors can be acknowledged as being reviewed by the forecaster. If all errors within the past 48 hours are acknowledged the system status icon at the lower right of the FEWS explorer will become green again.

With [Refresh] the browser is updated with new messages in the system. Please note that the query settings (see below) define what messages will be displayed. With [Close], a user can leave the Log Browser.

Upon opening, the same information is displayed as in the Log Viewer. Per message, the Log Browser provides the following information:

• Log level (see table above)
• Log creation time
• Event code (see Appendix A for listing)
• Log message (description)
• taskRunId

The following type of system messages are identified in the table below.

The Log Messages can be sorted by clicking on the header of a column.

Double click a message to view the message details (see example below). The message can then be saved to file or printed.

Query definition

A query can be defined to select a subset of the messages to be displayed in the Log Browser. The following options are available:

• Loglevel
  A log level from the dropdown list and all messages with a higher alert level will be displayed This means that for example selection of INFO shows all messages and selection of FATAL shows only messages with log level FATAL.

• Log creation time
  Messages can be displayed that have been created within a given period of time. The period is defined by entering an end time and the length of a view period in hours preceding the end time .

• Event code
  By entering symbols in this field, only messages will be displayed starting with the symbols entered by the user.
• Token in Message
  By entering symbols in this field, only messages will be displayed that include the symbols entered by the user.

The display will be updated on the basis of the user defined query upon pressing [Refresh].

Acknowledging messages

All messages having a level ERROR or FATAL should be acknowledged to identify that the message has been seen and appropriate action taken. 

A message having a level WARN can be acknowledged as well but is only needed to remove the opaque warning icon in the status bar if in the explorer.xml <showIconForUnacknowledgedWarnings>true</showIconForUnacknowledgedWarnings> is configured which is false by default. See showIconForUnacknowledgedWarnings.

Double click the icon to acknowledge a message or click the Acknowledge button. Multiple messages may also be selected and acknowledged in a single action.

To select all messages that have not been acknowledged select the 'not acknowledged' item in the Log Level selection box,, then refresh the log messages.
Tip: to quickly select all messages, click on a single message and hit [ctrl-A]

When all ERROR messages are acknowledged, the status icon in explorer will change to green (or opaque when showIconForUnacknowledgedWarnings is true) . 

Debug messages

Please note that a System Manager can configure the system to also provide Debug Messages . These are used to log application internal events for system development and management. Typical debug events are the creation of a database connection or exceptions that occurred due to programming or configuration errors. Debug messages are not displayed in the Log Viewer but only written to a log file called log.txt in the regional configuration folder.

Live System Status

The Live System Status tab is used to show the status of the live system. Three lists show the status of the:

• Currently dispatched forecasts (taskruns)
• Currently executing forecasts (taskruns)
• Status of the Master Controllers
• Status of the Forecasting Shells

The status lists show the currently configured master controllers and forecasting shells. From this list a user can see if the systems are Alive or if they are down.

Scheduled Forecasts

The Scheduled Forecasts tab includes a list with all scheduled forecasts together with some information of these forecasts. The following information is included:

• Task ID: The Task ID is a unique ID generated by the Master Controller
• Description The description is given by the user that scheduled the forecast
• Workflow ID The Workflow ID is the id of the workflow that is scheduled. The workflow id's can include Forecasting Shell workflows and Master controller workflows.
• MC ID ID of the Master Controller where the task is scheduled on
• Priority This can be a high or normal priority
• Repeat Time The repeat time of the task in minutes
• Next Due Time The next time when the task is scheduled to run
• Task Status The status can be suspended or pending. Suspended tasks normally depend on an action to trigger them, such an action can be a threshold crossing.

In the Scheduled Forecast tab a user can...

• Cancel a scheduled task by clicking [Cancel]
• Suspend a scheduled task by clicking [Suspend]
• Resume a suspended task by clicking [Resume]
• Pause a scheduled task for up to 5 of runs by clicking [Pause] and filling in the display that pops up (see below). You can undo the pausing by resetting the number of runs for which a task has been paused to '0'.

Please note that not all options are available for all workflows. A user can for example normally not cancel the current forecast and import runs runs scheduled by the super user. Also the status of system management workflows can normally not be changed by a user.

Running Forecasts

The Running Forecasts Tab includes a list with all tasks that are currently running on a Forecasting Shell. The list includes information on the task that is running and the Forecasting Shell it is running on.

Since 2021.02 this tab is also added to SA and it has 2 new columns:

• Activity progress: this shows the progress of the activity that is running, this will only show a number higher than 0 when execute activities in the general adapter either read the progress from the console of the executable and / or have <activityDurationWeight> configured. See the console element of the general adapter. 
• Workflow Node Index: this shows which part of the workflow is currently being executed. Via right mouse "show in workflow navigator" the workflow and its steps can be inspected. This way the user can track which part of the workflow is currently being exectuted.

A manually dispatched task can be killed by pressing on the 'Kill Running Task' button. Note that this is a hard process kill which may lead to corruption of the local datastore. 

Batch Forecasts

The Batch Forecast tab will list the forecasts that have been submitted in the Manual Forecast display of a standalone system. You stop all submitted forecast runs by clicking [Stop].

Synchronisation Status

The Synchronisation Status tab provides the status of all synchronisation activities. For each synchronisation activity the following information is provided:

• Activity: identifies the synchronisation activity
• Type: single or continuous. A single type activity is only started after certain manual actions
• Frequency: only applicable for continuous synchronisation activities. The frequency is given in minutes.
• Last Synch Time: the date and time the synchronisation activity has last taken place.
• Status: is either empty or shows 'synchronising', meaning that this activity is now synchronising.
• Progress: shows the number of records of the maximum number of records to synchronise. This field is only used for the activity being synchronised.

Import Status

The Import Status tab provides information on the files that were last imported. For each import task the file name of the last imported file is shown including the date and time of the import. Also the number of files that were imported in that task are shown, with the number of failed imports.

By default all data feeds are visible in the Import Status tab. If some data feeds are not wanted, for example because they are not really relevant, one can use an option disableDataFeedInfo in the TimeSeriesImport config file. Since 2019.02 obsolete data feeds can be deleted from the admin interface. See Files - Import Status. 

Last import time is updated only if at least one file has been successfully imported. The cell in the column ‘Files Failed’  is colored red when there is one or more files failed.

An example (see two pictures below) :

First picture shows fully successful imports  at 08:03:28. Second picture shows partly successful import (Level measured) and failed import (Discharge measured). The files has been imported  at 08:11:24. Last import time of the partly successful import has been updated. Last import time of the failed import is not updated, since no file has been successful  imported .

Last import time stays red if the time series has been never imported for that data feed. The picture below illustrates this.  There is no time series imported from Q_m.xml,  because the wanted time series were not available in the file. There is also no time series imported for Level measured, since the import folder was empty.