Table of Contents |
---|
Schema location: https://fewsdocs.deltares.nl/schemas/version1.0/performanceIndicatorSets.xsd
...
Performance Indicator module
The performance indicator module is used as an analysis tool in DELFT-FEWS to establish an overview of how well the forecasting system is performing in terms of accuracy of the individual forecasting module or in terms of the forecasting system as a whole. Performance can be assessed in two ways;
...
The second type of measure can be assessed once observed data for which forecasts were made becomes available.
...
...
Assessing performance of modules
The first and most simple application of the performance indicator module is in the traditional module calibration. This is by comparing two time series where one time series is the estimated series and the other is the reference time series. These time series are compared over a configurable length. As with other time series this is referenced with respect to the forecast start time (T0).
...
On establishing the performance, the indicator is returned as a time series (simulated historical). This time series is a non-equidistant time series, labelled as a forecast historical with the time stamp set to T01.1.1 Anchor
Assessing performance of forecast values- lead time accuracy
Performance of forecast is assessed on the basis of lead time accuracy. This is done by comparing the forecast lead time value against the observed value at the same time (received later!). For each lead time, this value is assessed over a given number of forecasts.
...
Lead time accuracy is evaluated using again the BIAS, MSE, or MAE or BIAS:
Lead time accuracy in Mean Square Error bias (mean error) LEAD_MSEBIAS
Lead time accuracy in bias Mean Square Error LEAD_BIASMSE
Lead time accuracy in Mean absolute error LEAD_MAE
where
is the lead time accuracy at time , J is the number of forecasts considered, is the reference value at time and is the estimated value at time ....
On selecting reference values 1.1.2 Anchor
Assessing performance of forecast values- timing of thresholds
An important indicator of performance is the timing of predicted threshold event crossings. Again this is evaluated over a number of forecasts. To evaluate this the threshold crossings in the indicator and the reference series are considered. For each pair of matching thresholds (matched on threshold id's) the time between the two is evaluated, and expressed either as a time bias (T_BIAS) or a time absolute error (T_MAE). Times are evaluated in terms of seconds.
...
The results of the evaluation are written as a time series (simulated historical), with as a reference time the T0 of the evaluation run and a time stamp for each
.Assessing performance of forecast forecast precipitation
Performance indicators available:
- Root Mean Square Error
- Root Mean Square Factor
- Bias
(information to be added)
Assessing performance of forecast peak accuracy
(information to be added)1.1.3
1.1.4 h3. Anchor
Configuration of performance module
1.1.5 Figure 134 Elements of the performance module configuration 1.1.6
1.1.7 h5.
performanceIndicatorSet
Root element for configuration of a performance Module indicator. Multiple elements may be defined for each performance indicator to be assessed.
...
- performanceIndicatorId : Optional Id for the configuration. Used for reference purposes only 1.1.8
1.1.9 h5.
inputVariable
Definition of inputVariables (time series). Input variables are identified by their VariableId. See transformation module on definition of the inputVariable element. An input variable will need to be defined for both simulated and for observed time series.
...
outputVariable
Definition of outputVariable time series of performance indicator values is to be written to. This will normally be a non-equidistant time series as it is not a-priori certain when the performance indicator module is run.
...
modulePerformanceIndicator
Root element for configuration of performance indicator assessing module performance
Attributes;
- indicatorType : selection of performance indicator. Enumeration of options includes;:
- bias
- meanabsoluteerror
- meansquareerror
- nashsutcliffeefficiency
- peakmeansquareerror
- volumeerror
- calculatedVariableId : VariableId to identify calculated time series
- observedVariableId : VariableId to identify observed (reference) time series
- outputVariableId : VariableId to write resulting Performance index time series to.
...
- sampleOutputVariableId : VariableId to write total number of values (samples) used for the analysis to the output time series. Optional.
- analysedCalculatedVariableId : VariableId to write exactly that part of the calculated series, which was used for the analysis, to the output series. Optional .
- analysedObservedVariableId : VariableId to write exactly that part of the observed series, which was used for the analysis, to the output series. Optional .
leadTimeAccuracyIndicator
Root element for configuration of performance indicator assessing lead time accuracy
...
- indicatorType : selection of performance indicator. Enumeration of options includes;
- bias
- meanabsoluteerror
- meansquareerror
- calculatedVariableId : VariableId to identify calculated time series
- observedVariableId : VariableId to identify observed (reference) time series
- outputVariableId : VariableId to write resulting Performance index time series to
- sampleOutputVariableId : VariableId to write total number of values (samples) used for the analysis to the output time series. Optional.
1.1.16
1.1.17 h5. NOTE: this option is not supported by all assessment types. In that case, there will be a nullpointer error. - intermediateValuesVariableId : VariableId to write intermediate values (as visible in log file if debug is on) to the output series. Optional and only applicable if leadTimePeriods are configured.
- analysedCalculatedVariableId : VariableId to write exactly that part of the calculated series, which was used for the analysis, to the output series. Optional and only applicable if leadTimePeriods are configured.
- analysedObservedVariableId : VariableId to write exactly that part of the observed series, which was used for the analysis, to the output series. Optional and only applicable if leadTimePeriods are configured.
thresholdTimingIndicator
Root element for configuration of performance indicator assessing accuracy of threshold Timing
Attributes;
- indicatorType : selection of performance indicator. Enumeration of options includes;
- bias
- meanabsoluteerror
- meansquareerror
- calculatedVariableId : VariableId to identify calculated time series
- observedVariableId : VariableId to identify observed (reference) time series
- outputVariableId : VariableId to write resulting Performance index time series to.
1.1.18 Figure 135 Elements of the ModulePerformance configuration 1.1.19
1.1.20 h5.
additionalCriteria
Additional criteria identified in establishing performance indicators. Application depends on the performance indicator selected.
...
- Criteria : list of criteria that may be applied. Enumeration of options includes;
- minnumberofforecasts
- timewindowinseconds
- thresholdvaluesetid
- peakthresholdvalue
- maximumgapbetweenpeaksinseconds
- minimumrecessionbetweenpeaks
- value: value of criteria defined
- violationOfCriteriaFlaggedAs: optional flag applied to PerformanceIndicator output series if criteria identified (eg. minnumberofforecasts )__ do not hold. Enumeration of;
- unreliable
- doubtful 1.1.21
1.1.22 h5.
description
Description of criteria defined. For reference purposes only.
...
...
Figure 136 Elements of the leadTimeAccuracy configuration
...
.
...
...
leadTimes
Root element for defining lead times
...
.
...
...
leadTime
Lead time for which to assess lead time performance.
Attributes;
- time : lead time in number
...
- of time units from the forecast time of the calculated time series. The time units are configured with the attribute 'unit'
- outputVariableId: variableId to output lead time accuracy to. This is defined when a separate time series is defined to keep track of performance at different lead times. It is not required when keeping track of performance in a single time series (Note that in the former a
...
- simulated historical time series can be used. In the latter this must be a simulated forecasting time series).
...
leadTimePeriods
Root element for defining lead time periods
leadTimePeriod
Lead time period for which to assess lead time performance.
Attributes;
- start : start of the period in number of time units from the forecast time of the calculated time series. The time units are configured with the attribute 'unit'
- end : end of the period in number of time units from the forecast time of the calculated time series. The time units are configured with the attribute 'unit'
- outputVariableId: variableId to output lead time accuracy to. This is defined when a separate time series is defined to keep track of performance at different lead time periods. It is not required when keeping track of performance in a single time series
- sampleOutputVariableId : variableId to output number of samples used in this lead time period. This is defined when a separate time series is defined to keep track of performance at different lead time periods. It is not required when keeping track of performance in a single time series
Figure 137 Elements of the thresholdTimingAccuracy configuration.
thresholdIds
Root element for defining threshold crossings to be assessed. 1.1.31
1.1.32 h5.
thresholdId
Configuration of threshold crossing to be checked.
...
- intId : Id of the threshold. See thresholds configuration in Regional Configuration.
- outputVariableId: variableId to output threshold timing accuracy to. This is defined when a separate time series is defined to keep track of performance for different thresholds. It is not required when keeping track of performance in a single time series (Note that in the former a simulate simulated historical time series can be used. In the latter this must be a simulated forecasting time series).
1.1.33