Page History

...

The SecondaryValidation module can be used to perform certain checks on time series data and generate log messages when the specified criteria are met.

Configuration

An XML file for configuring an instance of the SecondaryValidation module called for example CheckImportedData would be the following:

...

A SecondaryValidation configuration file is typically located in the moduleconfigfiles ModuleConfigFiles folder and can be used to configure one or more checks. The configured checks will be processed one by one in the specified order. Some The checks can generate log messages, which can trigger actions in the master controller, like e.g. sending warning e-mails. Another A special type of check is available for automatically modifying flags to 'doubtful' or 'unreliable' per time step when a constraint on multiple time series fails.

Checks for

...

counting reliable, doubtful, unreliable and missing values

These Some checks are intended for generating log events when a specific constraint is violated. The time series configured in these checks will be processed one by one. If a time series does not pass the check, then the configured log message is logged with the specified event code and level. The log event code can be used to trigger a certain action in the master controller, e.g. sending warning emails.

Four The following different types of these checks are available:

minNumberOfValuesCheck: Logs a message when there are not enough values within a configured period.
minNonMissingValuesCheck: Logs a message when there are not enough non-missing values within a configured period. A non-missing value is a value that is reliable, doubtful or unreliable.
minReliableOrDoubtfulValuesCheck: Logs a message when there are not enough values that are reliable or doubtful within a configured period.
minReliableValuesCheck: Logs a message when there are not enough reliable values within a configured period.

...

The seriesComparisonCheck check is available for testing constraints between multiple time series or parameters per time step.
This check verifies constraints between multiple time series sets or multiple parameters and automatically modifies the flags per time step when the required input data was available (reliable or doubtful) and the specified constraint failssucceeds.

Variable Definitions

The configuration contains variable definitions for one or more time series that can be used as input for checks. Each variable definition contains a variableId and a timeSeriesSet. The variableId can be used to reference the time series in a check. Alternatively, depending on which check it is, either variable definitions or variables can be embedded in the checks.

Contents of checks for generating

...

counting reliable, doubtful, unreliable and missing values

The minNumberOfValuesCheck, minNonMissingValuesCheck, minReliableOrDoubtfulValuesCheck and minReliableValuesCheck all consist of the following elements:

...

id: identifier of the check.
variableDefinition: embedded variable definition (see above).
checkRelativePeriod: The check will only consider data in this time period. This time period is relative to the timeZero of the taskrun in which the module instance runs. The start and end of the period are included. This period overrules any relativeViewPeriods specified in the timeSeriesSets of the time series.
expression: A comparison between one or expression: A comparison between one or more variableIds (see examples below).
validatingVariableId: One or more identifiers for variables for which the flags have to be modified.
outputFlag: New flag value for time steps for which there is valid data and the expression fails. Either doubtful or unreliable.
logLevel: Log level for the log message that is logged if a time series does not pass the check. Can be DEBUG, INFO, WARN, ERROR or FATAL. If level is error or fatal, then the module will stop running after logging the first log message. Fatal should never be used actually.
logEventCode: Event code for the log message that is logged if a time series does not pass the check. This event code has to contain a dot, e.g. "TimeSeries.Check", because the log message is only visible to the master controller if the event code contains a dot.
logMessage: Log message that is logged if a time series does not pass the check. Some more options are available than in the other checks:

Tag	Replacement
%AMOUNT_CHANGED_FLAGS%	The number of flags that has been altered.
%OUTPUT_FLAG%	The flag that has been set.
%CHECK_ID%	The id of the check that caused the flags to be altered.
%EXPRESSION%	The expression that caused the flags to be altered.
%LOCATION_ID%	The locationId where the alterations took place.
%PARAMETER_ID%	The parameterId where the alterations took place.

It is It is not possible to compare two different location sets both containing more than one location id, but the following comparisons can be configured:

...

Code Block

	xml
	xml

<?xml version="1.0" encoding="UTF-8"?>
<secondaryValidation xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/secondaryValidation.xsd">
    <!-- comparison between location variable and scalar, set to unreliable -->
	<seriesComparisonCheck id="checkWithScalar">
		<variableDefinition>
			<variableId>H_obs_location1</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheck</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs</parameterId>
				<locationId>location1</locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>
   		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<expression>H_obs_location1 .ge. 13</expression>
		<validatingVariableId>H_obs_location1</validatingVariableId>
		<outputFlag>unreliable</outputFlag>"checkWithScalar">
		<variableDefinition>
			<variableId>H_obs_location1</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheck</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs</parameterId>
				<locationId>location1</locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<relativeViewPeriod unit="day" start="-30" end="0"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>
		<expression>H_obs_location1 .ge. 13</expression>
		<validatingVariableId>H_obs_location1</validatingVariableId>
		<outputFlag>unreliable</outputFlag>
		<logLevel>INFO</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>%AMOUNT_CHANGED_FLAGS% flags set to %OUTPUT_FLAG% by [%CHECK_ID%, %EXPRESSION%].</logMessage>
    </seriesComparisonCheck>
</secondaryValidation>

...

Code Block

	xml
	xml

<?xml version="1.0" encoding="UTF-8"?>
<secondaryValidation xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/secondaryValidation.xsd">

<!-- comparison of variables with similar location sets, different parameters, does comparison per location  -->
	<seriesComparisonCheck id="similarLocationSetSeriesComparisonCheck">
		<!-- referred to by locationset1 and locationset2-->
		<variableDefinition>
			<variableId>H_obs1_location1</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheckTest</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs1</parameterId>
				<locationId>location1</locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<relativeViewPeriod unit="minuteday" start="-30" multiplierend="150"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>

		<!-- referred to by locationset1 and locationset2-->
		<variableDefinition>
			<variableId>H_obs1_location2</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheckTest</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs1</parameterId>
				<locationId>location2</locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<relativeViewPeriod unit="day" start="-30" end="0"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>
		<!-- referred to by locationset1 and locationset2-->
		<variableDefinition>
			<variableId>H_obs2_location1</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheckTest</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs2</parameterId>
				<locationId>location1</locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<relativeViewPeriod unit="minuteday" start="-30" multiplierend="150"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>
		<!-- referred to by locationset1 and locationset2-->
		<variableDefinition>
			<variableId>H_obs2_location2</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheckTest</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs2</parameterId>
				<locationId>location2</locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<relativeViewPeriod unit="day" start="-30" end="0"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>

		<variableDefinition>
			<variableId>locationSet1</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheckTest</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs</parameterId>
				<locationSetId>locationset1</locationSetId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<timeStep unit<relativeViewPeriod unit="day" start="minute-30" multiplierend="150"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</variableDefinition>

		<variableDefinition>
			<variableId>locationSet2</variableId>
			<timeSeriesSet>
				<moduleInstanceId>SeriesComparisonCheckTest</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>H.obs</parameterId>
				<locationSetId>locationset2</locationSetId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="minute" multiplier="15"/>
				<readWriteMode>read only</readWriteMode>
<relativeViewPeriod unit="day" start="-30" end="0"/>
				<<readWriteMode>read only</timeSeriesSet>readWriteMode>
			</variableDefinition>

		<checkRelativePeriod unit="hour" start="-12" end="0"/>/timeSeriesSet>
		</variableDefinition>

		<expression>abs(locationSet1 - locationSet2) .legt. 3</expression>
		<validatingVariableId>locationSet1</validatingVariableId>
		<validatingVariableId>locationSet2</validatingVariableId>
		<outputFlag>unreliable</outputFlag>
		<logLevel>INFO</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>%AMOUNT_CHANGED_FLAGS% flags set to %OUTPUT_FLAG% by %CHECK_ID%.</logMessage>
	</seriesComparisonCheck>
</secondaryValidation>

...

Page tree

Versions Compared

Old Version 20

New Version 21

Key

Configuration

Checks for

counting reliable, doubtful, unreliable and missing values

Variable Definitions

Contents of checks for generating

counting reliable, doubtful, unreliable and missing values