09 Thresholds

DELFT-FEWS supports checking of time series against thresholds. When thresholds are crossed, appropriate messages may be issued. Thresholds are defined in two files: thresholds.xml (described here) and thresholdValueSets.xml (described here). The first file contains definitions of threshold groups with thresholds and the types of threshold. There are three types of thresholds: level crossing thresholds, rate of rise and fall thresholds and max value thresholds. For a given threshold the options that are independent of location and parameter are defined.  Additional threshold options for specific locations and parameters are defined in the thresholdValueSets configuration file.  It is possible to configure one or more threshold groups.

Contents

General configuration options for threshold groups

eventExpiryTime: Optional expiry time for threshold events in the ThresholdEvent table in the database. If not configured the default expiry time of the system is applied to the threshold events.

maxActionEventDuration: Optional maximum duration of actions (like e.g. enhanced forecasting mode). An action can be switched on by a threshold upActionLogEvent (see thresholdValueSets). If the last upActionLogEvent issue is older than this period, then the action is automatically switched off. If not defined, defaults to 1 day.

forecastAvailableThreshold: Optional threshold that triggers forecast available threshold events. If this is specified, then forecast available events will be logged and these events can then be used in the SkillScoreDisplay. A forecast available threshold event indicates if there is a forecast available for a given observed threshold crossing event.

ThresholdGroup

A threshold group is a collection of one or more thresholds. The groups can be used to apply multiple threshold for displaying in graphs or generating reports. Each group can contain any number of level thresholds, rate thresholds and max value thresholds. For each threshold group it is possible to define a default threshold. The default provides extra information for the state in which no thresholds are crossed. The state in which no thresholds are crossed always corresponds to the least severe warning level in the thresholdWarningLevels configuration file. The least severe warning level is used e.g. to get the color to display in the user interface when no thresholds are crossed.

Level Threshold

A threshold that triggers a threshold event if a certain level is crossed upwards or downwards. For this type of threshold it is possible to configure an upWarningLevelId or a downWarningLevelId. There are also options to configure upIntId and downIntId, both of which are deprecated and should not be used when making new configurations.

Available Attributes:

id: Identifier of this threshold. Each threshold must have a unique id. Also thresholds that are in different threshold groups cannot have the same id.

name: Name of this threshold. This name is used when this threshold is displayed (e.g. in the timeSeriesDialog). If not specified, then the id is used when this threshold is displayed.

shortName: Optional short name for this threshold. This short name is used e.g. for the column headers in the threshold crossing counts tables in the ThresholdOverviewDisplay. If not specified, then the id is used as short name.

Available Elements:

upWarningLevelId:  Optional id of a warning level that will be activated for a particular location and parameter if a data value is above this threshold.

downWarningLevelId: Optional id of a warning level that will be activated for a particular location and parameter if a data value is below this threshold.

Warning levels are defined in the thresholdWarningLevels configuration file. If no warningLevels are defined, then this threshold will not activate any warningLevels to be displayed, but it will still generate threshold crossing events for both upward and downward crossings.

upIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed upwards. This id must be unique through this file. If not specified, then this id will be generated automatically.

downIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed downwards.

Similar to the Id's used, a warning level integer can be assigned to threshold crossings. This warning level is resolved to either an icon (for display in the main FEWS GUI), or a colour (for use in reports or schematic status displays). Warning levels need not be unique. If not specified this id will be generated automatically. Internally DELFT-FEWS maintains threshold events as a non-equidistant time series, where the crossings are identified by an integer. For each threshold two unique integer Id's need to be assigned. One ID is used to identify the upcrossing of the threshold, the other Id is assigned to identify the downcrossing. The exception to this is the peak threshold where only a single Id needs to be assigned to identify the occurrence of the peak event. Note: in the new thresholds configuration approach (thresholdGroups) these ids are optional and will be generated when not specified in configuration.

Rate Threshold

A threshold that triggers a threshold event if the rate of change of data values becomes higher or lower than a certain rate, e.g. -0.1 m per hour.

Available Attributes:

id: Identifier of this threshold. Each threshold must have a unique id. Also thresholds that are in different threshold groups cannot have the same id.

name: Name of this threshold. This name is used when this threshold is displayed (e.g. in the timeSeriesDialog). If not specified, then the id is used when this threshold is displayed.

Available Elements:

upIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed upwards. This id must be unique through this file. If not specified, then this id will be generated automatically.

downIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed downwards.

See level thresholds for more information about the up and down ids.

Max Threshold

A threshold that triggers a threshold event if there is a peak (maximum) in the data. The peak has to be higher than a certain level and has to be the maximum within a certain time window. If a maximum is at the end of a time series, then it does not count as a peak.

Available Attributes:

id: Identifier of this threshold. Each threshold must have a unique id. Also thresholds that are in different threshold groups cannot have the same id.

name: Name of this threshold. This name is used when this threshold is displayed (e.g. in the timeSeriesDialog). If not specified, then the id is used when this threshold is displayed.

Available Elements:

intId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is exceeded.  This id must be unique through this file. If not specified, then this id will be generated automatically.

See level thresholds for more information about the up and down ids.

Schema Thresholds.xsd

Level Threshold definition

Rate Threshold definition

Max Threshold definition

Explanation how threshold configuration is linked together

The ThresholdValueSets file contains threshold values for time series sets (e.g. level 3.2m  time series of for location A and parameter H.obs). This value is linked to a threshold group and to a threshold warning level via the (level)thresholdId. Each warning level corresponds to a unique integer that is called the severity of the warning level. Also see the figure below.

Explanation how threshold event logging is handled

Firstly to trigger the threshold crossing module, it is necessary to configure the predefinedActivity "threshold even crossing". When the threshold crossing module runs it will look at the configured timeseriesset in the ThresholdValueSets file and check if any data was written to these timeseries. Timeseries are only checked for threshold crossings if the data values have changed. For external timeseries it is possible to skip the check for changes and always check for threshold crossings. This can be configured with option 'onlyCheckThresholdsOfChangedSeries' in the workflow descriptor file.

 Once a threshold crossing check has been triggered the threshold event is logged. The logging process works as follows:

0) On initialization of the Threshold Crossing Module a subset of events is read from the ThresholdEventsTable. The number of records read is defined by the configuration parameter 'maxActionEventDuraion' times 10. Events older than this are considered irrelevant.

1) Check if  a newly logged event is already in the list of logged events retrieved from the database. For this check the following values are compared in the given order:

check threshold type
check threshold direction
check time series type 
check event time
check event value. Difference has a allowed tolerance of 10% of value.
For forecasted timeseries check the taskrunid
check moduleInstanceId
check thresholdValueSetId
check thresholdId
check locationId
check parameterId
check eventActionId
check rateOfChange value

2) If the new event is found in the list, it is discarded. If not it is added to the list of new events.

3) Once the threshold check has completed its inspection, the newly added events will be flushed.

4) On flushing the events are checked for 'action events'. Action Events cause System Alerter to trigger a configurable action. Logging of Action Events follows the following rules:
    - For none forecast timeseries, if a down action is logged after an up action for the same actionId then the Action Event is discarded.
    - For forecast timeseries each up action causes an Action Event. The forecast down action events are ignored.

   At the end of this action we have a list of all currently active Action Events.

5) Go over the list of currently active Action Events and check if they have exceeded their max action duration time. This is the maximum time that an action event can be active. This value is configured in the Thresholds file with element 'maxActionEventDuration'. Default value is 1 Day. If an up action has expired then a corresponding down Action Event needs to be created and added to the issue list. Action Events of none approved forecast timeseries are skipped.

6) Now we have a complete list of active Action Events we go through the list and set their 'issue' status. Setting the issue status to 'true' will cause the event to be logged so the System Alerter will trigger an action. Setting the issue status to 'false' will only result in the event being logged. The issued status is set to 'true' for the following cases:
    - Up crossing Action Events from the currently active list as long as they are not generated by none approved forecasts.
    - Down crossing Action Events as long as there is not a corresponding down Action Event active.

7) The list of active Action Events has be compiled. Down crossing Action Events have been created for expired up crossings. All 'issue' status values have been set. Now the events are logged. Depending on the events the following logs can be created:

With Action ID:
  Issued status 'true'
    Is up crossing
      (WARN) "<event action id>: UpCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      (WARN) "<event action id>: RateOfRise event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      (WARN) "<event action id>: PeakEvent event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
    Is down crossing
      Actual down crossing
        (WARN) "<event action id>: DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
        (WARN) "<event action id>: RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      Result of expired event
        (WARN) "<event action id>: Expired DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
        (WARN) "<event action id>: Expired RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
  Issued status 'false'
    Is up crossing
      (INFO) "<event action id>(not_issued): UpCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      (INFO) "<event action id>(not_issued): RateOfRise event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      (INFO) "<event action id>(not_issued): PeakEvent event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
    Is down crossing
      Actual down crossing
        (INFO) "<event action id>(not_issued): DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
        (INFO) "<event action id>(not_issued): RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      Result of expired event
        (INFO) "<event action id>(not_issued): Expired DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
        (INFO) "<event action id>(not_issued): Expired RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"

No Action ID:
    Is up crossing
      (DEBUG) "Threshold.Crossing: UpCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      (DEBUG) "Threshold.Crossing: RateOfRise event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      (DEBUG) "Threshold.Crossing: PeakEvent event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
    Is down crossing
      Actual down crossing
        (DEBUG) "Threshold.Crossing: DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
        (DEBUG) "Threshold.Crossing: RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
      Result of expired event
        (DEBUG) "Threshold.Crossing: Expired DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
        (DEBUG) "Threshold.Crossing: Expired RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"

Determination of most severe activated warning level

The most severe activated warning level is used for the warning icons and colours in the user interface and in the reports. Delft-FEWS takes the following steps to determine the most severe activated warning level for a given time series (the threshold log events are generated in a different but similar way).

1. First Delft-FEWS finds the thresholdValueSet (V) that corresponds to the given time series. If there is no thresholdValueSet defined that corresponds to the given time series, then no warning levels are activated, i.e. "All clear".
2. For the given time series only the data within a given time period is used. The TimeSeriesDialog and DataEditor use the period that is currently visible in the chart. The explorer user interface uses the relativeViewPeriod defined for the timeSeriesSet in the Filters configuration file. The ThresholdEventCrossingModule uses the relativeViewPeriod defined for the timeSeriesSet in the ThresholdValueSets configuration file. The ThresholdOverviewDisplay uses the configured aggregationTimeStep or relativePeriod in the ThresholdOverviewDisplay configuration file. Please note that in the ThresholdOverviewDisplay and in the Reports the data is read using the timeSeriesSets configured in the inputVariables. Therefore the relativeViewPeriods defined for the timeSeriesSets of the inputVariables must include the relativePeriod for which the most severe activated warning level has to be determined. Otherwise not all of the required data is read.
3. If the given time series contains only missing values, then no warning levels are activated, i.e. "All clear".
4. For each data value separately, Delft-FEWS considers each levelThresholdValue in V and determines if it has been crossed for the given data value (see above for definitions of crossed). Each levelThresholdValue that has been crossed, activates its corresponding warning level. From all the warning levels that are activated for the given data value, the most severe warning level is chosen. This is repeated for each data value within the given timeDisplay of thresholds in TimeSeriesDisplay

Configuring display options for the time series dialog

It is possible to define a custom color and linestyle for every threshold. The file TimeSeriesDisplayConfig contains a specific element, thresholdDisplayConfig, in which for every threshold a linestyle and color can be defined. More information on how to configure the thresholdDisplayConfig can be found at 02 Time Series Display Configuration, under ThresholdDisplayConfig. 

Note that if no color is configured for a threshold, the time series dialog option to show the threshold colors in the background will be silently disabled (the user can still toggle the option, but doing so will have no effect, the background will remain white). Therefore, it is advisable to always configure colors for the thresholds. 

Colors to be used can be found here.