What | Thresholds.xml |
---|---|
Required | no |
Description | definition of threshold groups. contains all threshold options that do not relate to locations or parameters. |
schema location | https://fewsdocs.deltares.nl/schemas/version1.0/thresholds.xsd |
DELFT-FEWS supports checking of time series against thresholds. When thresholds are crossed, appropriate messages may be issued. Thresholds are defined in two type of files: thresholds.xml (described here) and thresholdValueSets.xml (described here). The first file contains definitions of threshold groups with thresholds and the types of threshold. There are three types of thresholds: level crossing thresholds, rate of rise and fall thresholds and max value thresholds. For a given threshold the options that are independent of location and parameter are defined. Additional threshold options for specific locations and parameters are defined in the thresholdValueSets configuration file. It is possible to configure one or more threshold groups. It is possible to configure threshold groups through several files. Note that the general configuration options should be either defined only once, or defined exactly in the same way in each file.
Contents
General configuration options for threshold groups
eventExpiryTime: Optional expiry time for threshold events in the ThresholdEvent table in the database. If not configured the default expiry time of the system is applied to the threshold events.
maxActionEventDuration: Optional maximum duration of actions (like e.g. enhanced forecasting mode). An action can be switched on by a threshold upActionLogEvent (see thresholdValueSets). If the last upActionLogEvent issue is older than this period, then the action is automatically switched off. If not defined, defaults to 1 day.
forecastAvailableThreshold: Optional threshold that triggers forecast available threshold events. If this is specified, then forecast available events will be logged and these events can then be used in the SkillScoreDisplay. A forecast available threshold event indicates if there is a forecast available for a given observed threshold crossing event.
ThresholdGroup
A threshold group is a collection of one or more thresholds. The groups can be used to apply multiple threshold for displaying in graphs or generating reports. Each group can contain any number of level thresholds, rate thresholds and max value thresholds. For each threshold group it is possible to define a default threshold. The default provides extra information for the state in which no thresholds are crossed. The state in which no thresholds are crossed always corresponds to the least severe warning level in the thresholdWarningLevels configuration file. The least severe warning level is used e.g. to get the color to display in the user interface when no thresholds are crossed.
Level Threshold
A threshold that triggers a threshold event if a certain level is crossed upwards or downwards. For this type of threshold it is possible to configure an upWarningLevelId or a downWarningLevelId. There are also options to configure upIntId and downIntId, both of which are deprecated and should not be used when making new configurations.
Available Attributes:
id: Identifier of this threshold. Each threshold must have a unique id. Also thresholds that are in different threshold groups cannot have the same id.
name: Name of this threshold. This name is used when this threshold is displayed (e.g. in the timeSeriesDialog). If not specified, then the id is used when this threshold is displayed.
shortName: Optional short name for this threshold. This short name is used e.g. for the column headers in the threshold crossing counts tables in the ThresholdOverviewDisplay. If not specified, then the id is used as short name.
Available Elements:
upWarningLevelId: Optional id of a warning level that will be activated for a particular location and parameter if a data value is above this threshold.
downWarningLevelId: Optional id of a warning level that will be activated for a particular location and parameter if a data value is below this threshold.
Warning levels are defined in the thresholdWarningLevels configuration file. If no warningLevels are defined, then this threshold will not activate any warningLevels to be displayed, but it will still generate threshold crossing events for both upward and downward crossings.
upIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed upwards. This id must be unique through this file. If not specified, then this id will be generated automatically.
downIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed downwards.
season: Optional to specify if a threshold is only valid for part of a year. When not configured the threshold is valid for the entire year.
Similar to the Id's used, a warning level integer can be assigned to threshold crossings. This warning level is resolved to either an icon (for display in the main FEWS GUI), or a colour (for use in reports or schematic status displays). Warning levels need not be unique. If not specified this id will be generated automatically. Internally DELFT-FEWS maintains threshold events as a non-equidistant time series, where the crossings are identified by an integer. For each threshold two unique integer Id's need to be assigned. One ID is used to identify the upcrossing of the threshold, the other Id is assigned to identify the downcrossing. The exception to this is the peak threshold where only a single Id needs to be assigned to identify the occurrence of the peak event. Note: in the new thresholds configuration approach (thresholdGroups) these ids are optional and will be generated when not specified in configuration.
Rate Threshold
A threshold that triggers a threshold event if the rate of change of data values becomes higher or lower than a certain rate, e.g. -0.1 m per hour.
Available Attributes:
id: Identifier of this threshold. Each threshold must have a unique id. Also thresholds that are in different threshold groups cannot have the same id.
name: Name of this threshold. This name is used when this threshold is displayed (e.g. in the timeSeriesDialog). If not specified, then the id is used when this threshold is displayed.
Available Elements:
upIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed upwards. This id must be unique through this file. If not specified, then this id will be generated automatically.
downIntId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is crossed downwards.
season: Optional to specify if a threshold is only valid for part of a year. When not configured the threshold is valid for the entire year.
See level thresholds for more information about the up and down ids.
Max Threshold
A threshold that triggers a threshold event if there is a peak (maximum) in the data. The peak has to be higher than a certain level and has to be the maximum within a certain time window. If a maximum is at the end of a time series, then it does not count as a peak.
Available Attributes:
id: Identifier of this threshold. Each threshold must have a unique id. Also thresholds that are in different threshold groups cannot have the same id.
name: Name of this threshold. This name is used when this threshold is displayed (e.g. in the timeSeriesDialog). If not specified, then the id is used when this threshold is displayed.
Available Elements:
intId: DEPRECATED, do not use in new configurations. Id that will be put to the output time series if a particular threshold value of this type is exceeded. This id must be unique through this file. If not specified, then this id will be generated automatically.
season: Optional to specify if a threshold is only valid for part of a year. When not configured the threshold is valid for the entire year.
See level thresholds for more information about the up and down ids.
Event Threshold
An event threshold is similar to a level threshold, only it has the possibility to be based on correlation events.
The advantage to this is that for instance peak heights that occured through the years can be used in the correlation display as well as a threshold without storing the same information in different ways.
<thresholdValueSet id="EventThresholds" name="EventThresholds"> <eventThresholdValues> <eventThresholdId>LevelWarn</eventThresholdId> <valueFunction>@PEAK_HEIGHT_VALUE@</valueFunction> <labelFunction>@PEAK_HEIGHT_TIME@</labelFunction> <parameterIdFunction>@PEAK_HEIGHT_PAR@</parameterIdFunction> <enabledFunction>@PEAK_HEIGHT_ENABLED@</enabledFunction> </eventThresholdValues> <timeSeriesSet> <moduleInstanceId>ThresholdCheck</moduleInstanceId> <valueType>scalar</valueType> <parameterId>H.m</parameterId> <locationSetId>MultiValuedAttributeThresholds</locationSetId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="hour" divider="1" multiplier="1"/> <readWriteMode>add originals</readWriteMode> </timeSeriesSet> </thresholdValueSet>
Schema Thresholds.xsd
Level Threshold definition
Rate Threshold definition
Max Threshold definition
ThresholdWarningLevels
Different threshold warning levels can be defined. They need to be in order of increasing severity. First level defined is always the no-threshold-cross level.
Optionally a missingDataThreshold can be defined. Setting a different cell background color for this allows to differentiate easier between no threshold crossing and no data available. If missingDataThreshold is defined it must precede the thresholdWarningLevels. Example:
Explanation how threshold configuration is linked together
The ThresholdValueSets file contains threshold values for time series sets (e.g. level 3.2m time series of for location A and parameter H.obs). This value is linked to a threshold group and to a threshold warning level via the (level)thresholdId. Each warning level corresponds to a unique integer that is called the severity of the warning level. Also see the figure below.
Explanation how threshold event logging is handled
Firstly to trigger the threshold crossing module, it is necessary to include the predefinedActivity "threshold event crossing" in a workflow. When the threshold crossing module runs it will look at the configured timeseriesset in the ThresholdValueSets file and check if any data was written to these timeseries. Timeseries are only checked for threshold crossings if the data values have changed. For external timeseries it is possible to skip the check for changes and always check for threshold crossings. This can be configured with option 'onlyCheckThresholdsOfChangedSeries' in the workflow descriptor file.
Once a threshold crossing check has been triggered the threshold event is logged in the database. Depending on issue status and event coupling, a message will appear in the log panel at a certain log level. The logging process works as follows:
0) On initialization of the Threshold Crossing Module a subset of events is read from the ThresholdEventsTable. The number of records read is defined by the configuration parameter 'maxActionEventDuraion' times 10. Events older than this are considered irrelevant.
1) Check if a newly logged event is already in the list of logged events retrieved from the database. For this check the following values are compared in the given order:
check threshold type check threshold direction check time series type check event time check event value. Difference has an allowed tolerance of 10% of value. For forecasted timeseries check the taskrunid check moduleInstanceId check thresholdValueSetId check thresholdId check locationId check parameterId check eventActionId check rateOfChange value
2) If the new event is found in the list, it is discarded. If not it is added to the list of new events.
3) Once the threshold check has completed its inspection, the newly added events will be flushed.
4) On flushing the events are checked for 'action events'. Action Events cause System Alerter to trigger a configurable action. Logging of Action Events follows the following rules:
- For none forecast timeseries, if a down action is logged after an up action for the same actionId then the Action Event is discarded.
- For forecast timeseries each up action causes an Action Event. The forecast down action events are ignored.
At the end of this action we have a list of all currently active Action Events. Threshold crossings for which no Action Event is configured in a 'LocationThresholds.csv' will not appear at WARN level in the log panel.
5) Go over the list of currently active Action Events and check if they have exceeded their max action duration time. This is the maximum time that an action event can be active. This value is configured in the Thresholds file with element 'maxActionEventDuration'. Default value is 1 Day. If an up action has expired then a corresponding down Action Event needs to be created and added to the issue list. Action Events of none approved forecast timeseries are skipped.
6) Now we have a complete list of active Action Events we go through the list and set their 'issue' status. Setting the issue status to 'true' will cause the event to be logged so the System Alerter will trigger an action. Setting the issue status to 'false' will only result in the event being logged. The issued status is set to 'true' for the following cases:
- Up crossing Action Events from the currently active list as long as they are not generated by none approved forecasts.
- Down crossing Action Events as long as there is not a corresponding down Action Event active.
7) The list of active Action Events has be compiled. Down crossing Action Events have been created for expired up crossings. All 'issue' status values have been set. Now the events are logged. Depending on the events the following logs can be created:
With Action ID: Issued status 'true' Is up crossing (WARN) "<event action id>: UpCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>: RateOfRise event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>: PeakEvent event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Is down crossing Actual down crossing (WARN) "<event action id>: DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>: RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Result of expired event (WARN) "<event action id>: Expired DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>: Expired RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Issued status 'false' Is up crossing (WARN) "<event action id>(not_issued): UpCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>(not_issued): RateOfRise event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>(not_issued): PeakEvent event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Is down crossing Actual down crossing (WARN) "<event action id>(not_issued): DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>(not_issued): RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Result of expired event (WARN) "<event action id>(not_issued): Expired DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (WARN) "<event action id>(not_issued): Expired RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" No Action ID: Is up crossing (DEBUG) "Threshold.Crossing: UpCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (DEBUG) "Threshold.Crossing: RateOfRise event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (DEBUG) "Threshold.Crossing: PeakEvent event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Is down crossing Actual down crossing (DEBUG) "Threshold.Crossing: DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (DEBUG) "Threshold.Crossing: RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" Result of expired event (DEBUG) "Threshold.Crossing: Expired DownCrossing event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>" (DEBUG) "Threshold.Crossing: Expired RateOfFall event <threshold name> for <parameterid> at <locationid> (<location name>). Time <date> <time>"
Determination of most severe activated warning level
The most severe activated warning level is used for the warning icons and colours in the user interface and in the reports. Delft-FEWS takes the following steps to determine the most severe activated warning level for a given time series (the threshold log events are generated in a different but similar way).
- First Delft-FEWS finds the thresholdValueSet (V) that corresponds to the given time series. If there is no thresholdValueSet defined that corresponds to the given time series, then no warning levels are activated, i.e. "All clear".
- For the given time series only the data within a given time period is used. The TimeSeriesDialog and DataEditor use the period that is currently visible in the chart. The explorer user interface uses the relativeViewPeriod defined for the timeSeriesSet in the Filters configuration file. The ThresholdEventCrossingModule uses the relativeViewPeriod defined for the timeSeriesSet in the ThresholdValueSets configuration file. The ThresholdOverviewDisplay uses the configured aggregationTimeStep or relativePeriod in the ThresholdOverviewDisplay configuration file. Please note that in the ThresholdOverviewDisplay and in the Reports the data is read using the timeSeriesSets configured in the inputVariables. Therefore the relativeViewPeriods defined for the timeSeriesSets of the inputVariables must include the relativePeriod for which the most severe activated warning level has to be determined. Otherwise not all of the required data is read.
- If the given time series contains only missing values, then no warning levels are activated, i.e. "All clear".
- For each data value separately, Delft-FEWS considers each levelThresholdValue in V and determines if it has been crossed for the given data value (see above for definitions of crossed). Each levelThresholdValue that has been crossed, activates its corresponding warning level. From all the warning levels that are activated for the given data value, the most severe warning level is chosen. This is repeated for each data value within the given timeDisplay of thresholds in TimeSeriesDisplay
Configuring display options for the time series dialog
It is possible to define a custom color and linestyle for every threshold. The file TimeSeriesDisplayConfig contains a specific element, thresholdDisplayConfig, in which for every threshold a linestyle and color can be defined. More information on how to configure the thresholdDisplayConfig can be found at 02 Time Series Display Configuration, under ThresholdDisplayConfig.
Note that if no color is configured for a threshold, the time series dialog option to show the threshold colors in the background will be silently disabled (the user can still toggle the option, but doing so will have no effect, the background will remain white). Therefore, it is advisable to always configure colors for the thresholds.
Colors to be used can be found here.
Location specific threshold icons
You can make the threshold icons specific for different locationSets, if you link thresholds (Thresholds.xml) to specific locationSets (ThresholdValueSets) and assign them specific icons (ThesholdWarningLevels). If you also give the state road locations a blank icon (locationIcons.xml), the threshold crossings really stand out on the map. Especially if you also disable the visibility of the validationIcons (Filters.xml). See below an example for state roads.
Run ThresholdEventCrossing module in workflow
Run Threshold Event Crossing module in workflow
The ThresholdEventCrossing module is replaced in 2013 by a predefined activity. This predefined activity has the same functionality as the old ThresholdEvent module. You can add the following predefined activity to your workflow:
<activity> <runIndependent>false</runIndependent> <predefinedActivity>threshold event crossing</predefinedActivity> </activity>
Do make sure to set the relativeViewPeriod correctly in ThresholdValueSets.xml:
<timeSeriesSet> <moduleInstanceId>ImportCSV</moduleInstanceId> <valueType>scalar</valueType> <parameterId>H.obs</parameterId> <locationId>Bhadra_Reservoir</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep id="daily0600"/> <relativeViewPeriod unit="day" start="-5" end="0"/> <readWriteMode>read only</readWriteMode> </timeSeriesSet>
For the threshold event values to be visible in the ThresholdEvents display the view period needs to be set:
<eventTimeViewPeriod start="-365" end="48" unit="day"/>
If this is not done it will default to -10 day to +10 days. This does not have to be a problem because it is always possible to overrule the view period in the display itself by mean of filtering: double-click 'Event Time' column and a pop-up will appear where the user can set the period for viewing threshold events.
When the predefined activity is added to the workflow. Please note:
- The module will take all external time series into account (external historical and external forecast series created inside and outside the workflow) and only simulated (simulated historical, simulated forecasting) time series which are created within the same workflow.
- When the module should only check external time series changed within the current workflow, the attribute "onlyCheckThresholdsOfChangedSeries" in the workflow descriptor file need to be set to true. This need to be added to all workflowdescriptors that have the threshold crossing module included.
- The module needs to be added at the end of the workflow where the timeseries have been created (e.g. imported or by transformation).
- The relativeViewPeriod configured for the timeseriesSets in the ThresholdValueSets.xml determines the viewPeriod for which thresholdcrossings are determined. If the timeseries that has been imported or otherwise created is longer than the configured relative view period, the values outside this relativeViewPeriod will not be validated.
- As a result, the threshold crossing logs are written to the database, and can for example be used in eventActions and shown in the system wide threshold events display.
- When time series from within a threshold value set should not be taken into account, do not use a <relativeViewPeriod> for them and not a <readWriteMode> that says "read complete forecast".
Seasonal Thresholds
Since 2018.02 thresholds can be made seasonal if a threshold is only valid for part of a year. When not configured the threshold is valid for the entire year.
Within a threshold group and a threshold value set, both seasonal and non-seasonal thresholds can be used together.
<thresholdGroup id="Waterlevel" name="Seasonal Thresholds"> <defaultThreshold shortName="None"/> <levelThreshold id="River Alert" name="River Alert"> <upWarningLevelId>River Alert</upWarningLevelId> </levelThreshold> <levelThreshold id="Minor Flooding June" name="Minor Flooding June"> <upWarningLevelId>Minor Flooding</upWarningLevelId> <season startMonthDay="--06-01" endMonthDay="--06-30"/> </levelThreshold> <levelThreshold id="Minor Flooding July" name="Minor Flooding July"> <upWarningLevelId>Minor Flooding</upWarningLevelId> <season startMonthDay="--07-01" endMonthDay="--07-31"/> </levelThreshold> <levelThreshold id="Minor Flooding August" name="Minor Flooding August"> <upWarningLevelId>Minor Flooding</upWarningLevelId> <season startMonthDay="--08-01" endMonthDay="--08-31"/> </levelThreshold> </thresholdGroup>
When a threshold is seasonal and drawn in the graph, the part of the threshold that is out of season is drawn in light grey to illustrate it is not valid. If the entire view period is out of season the theshold line will not be drawn at all.
If a threshold is crossed at system time and the system time is within the season of the threshold, the crossing will be taken into account.
If threshold icons are turned on in the explorer a warning icon will be shown in the data viewer:
If a threshold is crossed at system time and the system time is NOT within the season of the threshold, the crossing will not be taken into account.
This is illustrated in the next picture where no warning overlay icon will be drawn:
Grouping threshold labels (since 2020.02)
Since 2020.02 when multiple threshold labels belong to the same threshold id and have the same value, they will be grouped together and the threshold id plus the amount of thresholds will be shown between [ ].