Page History

...

Introduction

Validation rules are defined in DELFT-FEWS to allow quality checking of all time series data (scalar time series only). Several validation criteria may be defined per time series. All validation rules for all time series are defined in this configuration. For each time series to be checked, a set of validation rules is defined. Defining validation rules to apply to a time series set using a locationSet rather than identifying series individually can simplify the configuration greatly. Most validation rules may be defined either as a constant value, or as a value valid per calendar month.

...

For extremeValues, rateOfChange, sameReading and temporaryShift a locationId can be given to make the rule location specific. Before 2014.02 this functionality was used to make multiple location-specific rules of the same type within a single validationRuleSet.

From 2014.02 on multiple on multiple rules of the same type within a single validationRuleSet can be given to apply to all locations within the timeSeriesSet instead of them to be location specific. Also from 2014.02 on multiple rateOfChangeFunctionsmultiple rateOfChangeFunctions, sameReadingFunctions and temporaryShiftFunctions can be used within the same validation rule set.

Validation on extreme values

This group of validation rules checks that the values in the time series do not exceed minimum and maximum limits. These limits may be defined as soft limits or as hard limits. Values exceeding soft limits will be marked as doubtful but retained. Values exceeding hard limits will be marked as unreliable. In case any of these rules are violated, a violation comment is added to the time step. If the time step already includes a comment, the violation comment is appended to the original comment. If no violation comment is provided, then the original comments would be added in the output only.Image Removed
Figure 34 Elements of the Extreme values configuration of the ValidationRuleSets.

hardMax

Validation rule for checking hard maximum. Values exceeding this limit will be marked as unreliable.

...

Element used when defining variable limits per calendar month. Twelve values must be defined. When defined the monthly limit will overrule the constant limit.

Validation on rate of change

This group of validation rules checks that the values in the time series do not exceed maximum rates of change. When the rate of change limit is exceeded, the values causing the limit to be exceeded will be marked as unreliable. Rate of change limits may be defined to be the same for the rate of rise as for the rate of fall. These may also be defined to be different. The rates need to be specified in the unit of the timeseries it applies per second. E.g. if you define a rate of change for a water level gauge with values in metres the rate should be given in metres per second. Please note that this validationRule can only be succesfully applied to a value if there is a value at the time step before it as well. Especially when applying this rule to a temporary series that is merged (i.e. including the validation flags) to a permanent series this can lead to unwanted behaviour. The first value of the temporary series can't be validated with this rule, as there exists no value before it, even though there is in fact a value present in the destination series of the merge. 

...

Validation on series of same readings

Time series of data can be validated on series of same readings. This may be unlikely for field observations, and may indicate an instrumental error. In some cases a small variability may still be observed, despite instrumental error. The same readings check allows for defining a bandwidth within the value is considered to be the same.  Please Please note that this validationRule can only be succesfully applied to a value if there is a value at the time step before it as well. Especially when applying this rule to a temporary series that is merged (i.e. including the validation flags) to a permanent series this can lead to unwanted behaviour. The first value of the temporary series can't be validated with this rule, as there exists no value before it, even though there is in fact a value present in the destination series of the merge. Image Removed
Figure 36 Elements of the same reading configuration of the ValidationRuleSets.

SameReadingViolations are marked as such if values stay within the sameReadingDeviation for at least the sameReadingPeriod. The way the check is applied is by having each value in the timeSeries be considered as potential start of a period in which the reading will be the same. In the case of a violation, a violation comment is added to the time step. If the time step already includes a comment, the violation comment is appended to the original comment. 

...

Since 2020.01. If true a same reading period ends when a missing value is found. Default is false.

Validation on Temporary Shifts

Time series of data can be validated on temporary shifts. These occur when instruments are reset, and can be identified by the values rapidly falling to a constant value, remaining at that value for a short period of time and then returning to the original value range. A complex set of validation criteria include the rate of change as well as a maximum time the value remains the same. Please note that this validationRule can only be succesfully applied to a value if there is a value at the time step before it as well (or multiple). Especially when applying this rule to a temporary series that is merged (i.e. including the validation flags) to a permanent series this can lead to unwanted behaviour. The first value of the temporary series can't be validated with this rule, as there exists no value before it, even though there is in fact a value present in the destination series of the merge. 

...

Element used when defining variable limits per calendar month. Twelve values must be defined. When defined the monthly limit will overrule the constant limit.

Validation on Oscillation

Time series can be validated on whether oscillation occurs. Oscillation is recognized when the value rapidly falls and rises in an alternating pattern. This validation can also be used to only mark values where oscillation occurs with an oscillation flag source, but keep the original flag of the value. This can be useful since oscillation might for example occur due to an error in the automatic steering of a pump causing it to rapidly turn on/off. In this case, the oscillation can be detected, without marking the actual values as "doubtful" or "unreliable" since the measured value is correct.

...

Example for "Rate of Rise" and "Temporary Shift" validation rules

Use of <timeSeries> instead of <timeSeriesSets>

Since 2023.02 it is possible to use the <timeSeries> element instead of the <timeSeriesSets> elements.

...

	<validationRuleSet validationRuleSetId="SetWithTimeSeriesFilter" timeZone="GMT">
		<extremeValues>
			<hardMax constantLimit="20"/>
			<hardMin constantLimit="0"/>
			<softMax constantLimit="10"/>
			<softMin constantLimit="2"/>
		</extremeValues>
		<timeSeries>
			<moduleInstanceId>moduleA</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterGroupId>ParGroupA</parameterGroupId>
			<parameterId>Par1</parameterId>
			<parameterId>Par2</parameterId>
			<locationSetId>locationSetA</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour"/>
		</timeSeries>
	</validationRuleSet>

Examples of validation rules

Example for "Rate of Rise" and "Temporary Shift" validation rules

Changing validation 

When validation rules have changed after data has been already validated, it is possible to revalidate all time series connected to all validation rules via the revalidation module which is available from version 2023.02.

...