21 Secondary Validation

<moduleDescriptor id="SecondaryValidation">
	<description>SecondaryValidation</description>
	<className>nl.wldelft.fews.system.plugin.secondaryValidation.SecondaryValidation</className>
</moduleDescriptor>

Secondary validation (since build 25550)

The Secondary Validation module can be used to perform certain checks on time series data. Some checks can generate log messages, which can trigger actions in the master controller, like e.g. sending warning e-mails. Other checks may be used to modify flags per time step when a constraint in a specified expression fails.

Configuration

An XML file for configuring an instance of the SecondaryValidation module called for example CheckImportedData could contain the following:

CheckImportedData 1.00 default.xml

A SecondaryValidation configuration file is typically located in the moduleconfigfiles folder and can be used to specify one or more checks. The configured checks will be processed one by one in the specified order.

Checks for generating log events

Some checks are intended for generating log events when a specific constraint is violated. These checks will process the configured time series one by one. If a time series does not pass the check, then the configured log message is logged with the specified event code and level. The log event code can be used to trigger a certain action in the master controller, e.g. sending warning emails.

Four different types of these checks are available:

• minNumberOfValuesCheck: Checks whether there are enough values within a configured period, otherwise a log message is logged.
• minNonMissingValuesCheck: Checks whether there are enough non-missing values within a configured period, otherwise a log message is logged. A non-missing value is a value that is reliable, doubtful or unreliable.
• minReliableOrDoubtfulValuesCheck: Checks whether there are enough values that are reliable or doubtful within a configured period, otherwise a log message is logged.
• minReliableValuesCheck: Checks whether there are enough reliable values within a configured period, otherwise a log message is logged.

Checks for setting flags

The TimeSeriesComparisonCheck can verify constraints between multiple timeseries sets and automatically modify the flags per timestep when the specified constraint fails.

Variable Definitions

The configuration contains variable definitions for one or more time series that can be used as input for checks. Each variableDefinition contains a variableId and a timeSeriesSet. The variableId can be used to reference the time series in a check. Alternatively, depending on which check it is, either variable definitions or variables can be embedded in the checks.

Contents of Checks

The minNumberOfValuesCheck, minNonMissingValuesCheck, minReliableOrDoubtfulValuesCheck and minReliableValuesCheck all consist of the following elements:

• id: Identifier of the check. This is only used in log messages and exception messages.
• timeSeries: One or more time series that need to be checked. This can be either an embedded timeSeriesSet or a reference to a time series defined at the start of the configuration file. If this contains multiple time series (e.g. for multiple locations), then each time series is checked individually.
• checkRelativePeriod: The check will only consider data in this time period. This time period is relative to the timeZero of the taskrun in which the module instance runs. The start and end of the period are included. This period overrules any relativeViewPeriods specified in the timeSeriesSets of the time series.
• minNumberOfValues: The minimum required number of values in the time series to pass the check.
• logLevel: Log level for the log message that is logged if a time series does not pass the check. Can be DEBUG, INFO, WARN, ERROR or FATAL. If level is error or fatal, then the module will stop running after logging the first log message.
• logEventCode: Event code for the log message that is logged if a time series does not pass the check. This event code has to contain a dot, e.g. "TimeSeries.Check", because the log message is only visible to the master controller if the event code contains a dot.
• logMessage: Log message that is logged if a time series does not pass the check. It is possible to use the tag %HEADER% in the logMessage. The %HEADER% tag will be replaced with the header of the time series.

Configuration Example

A configuration example with four different checks configured is given below:

<timeSeriesAvailabilityChecker xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesAvailabilityChecker.xsd">
	<timeSeries>
		<timeSeriesId>input1</timeSeriesId>
		<timeSeriesSet>
			<moduleInstanceId>CheckImportedData</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>location1</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="15"/>
			<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</timeSeries>
	<timeSeries>
		<timeSeriesId>input2</timeSeriesId>
		<timeSeriesSet>
			<moduleInstanceId>CheckImportedData</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>location2</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="15"/>
			<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</timeSeries>
	<minReliableValuesCheck id="MinReliableValuesCheck">
		<timeSeries>
			<timeSeriesId>input1</timeSeriesId>
		</timeSeries>
		<timeSeries>
			<timeSeriesId>input2</timeSeriesId>
		</timeSeries>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>6</minNumberOfValues>
		<logLevel>WARN</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough reliable values available for time series %header%</logMessage>
	</minReliableValuesCheck>
	<minReliableOrDoubtfulValuesCheck id="MinReliableOrDoubtfulValuesCheck">
		<timeSeries>
			<timeSeriesId>input1</timeSeriesId>
		</timeSeries>
		<timeSeries>
			<timeSeriesId>input2</timeSeriesId>
		</timeSeries>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>12</minNumberOfValues>
		<logLevel>WARN</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough reliable or doubtful values available for time series %header%</logMessage>
	</minReliableOrDoubtfulValuesCheck>
	<minNonMissingValuesCheck id="MinNonMissingValuesCheck">
		<timeSeries>
			<timeSeriesId>input1</timeSeriesId>
		</timeSeries>
		<timeSeries>
			<timeSeriesId>input2</timeSeriesId>
		</timeSeries>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>18</minNumberOfValues>
		<logLevel>INFO</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough non-missing values available for time series %header%</logMessage>
	</minNonMissingValuesCheck>
	<minNumberOfValuesCheck id="MinNumberOfValuesCheck">
		<timeSeries>
			<timeSeriesSet>
				<moduleInstanceId>CheckImportedData</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>P.obs</parameterId>
				<locationSetId>locationSet1</locationSetId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="nonequidistant"/>
				<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</timeSeries>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>24</minNumberOfValues>
		<logLevel>DEBUG</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough values available for time series %header%</logMessage>
	</minNumberOfValuesCheck>
</timeSeriesAvailabilityChecker>