Page History

...

What

nameofinstance.xml

Description

Configuration for the Secondary Validation module

schema location

http://fews.wldelft.nl/schemas/version1.0/secondaryValidation.xsd

Entry in ModuleDescriptors

Code Block

	xml
	xml

<moduleDescriptor id="SecondaryValidation">
	<description>SecondaryValidation</description>
	<className>nl.wldelft.fews.system.plugin.secondaryValidation.SecondaryValidation</className>
</moduleDescriptor>

Secondary Validation SecondaryValidation (since 2010_01)

The Secondary Validation SecondaryValidation module can be used to perform certain checks on time series data. Some checks can generate log messages, which can trigger actions in the master controller, like e.g. sending warning e-mails. Another check is available for automatically modifying flags to 'doubtful' or 'unreliable' per time step when a constraint on multiple time series fails.

...

The configuration contains variable definitions for one or more time series that can be used as input for checks. Each variableDefinition variable definition contains a variableId and a timeSeriesSet. The variableId can be used to reference the time series in a check. Alternatively, depending on which check it is, either variable definitions or variables can be embedded in the checks.

Contents of

...

checks for generating log events

The minNumberOfValuesCheck, minNonMissingValuesCheck, minReliableOrDoubtfulValuesCheck and minReliableValuesCheck all consist of the following elements:

id: Identifier of the check. This is only used in log messages and exception messages.
timeSeriesvariable: One or more time series that need to be checked. This can be either an embedded timeSeriesSet or a reference to a time series variabledDefinition defined at the start of the configuration file. If this contains multiple time series (e.g. for multiple locations), then each time series is checked individually.
checkRelativePeriod: The check will only consider data in this time period. This time period is relative to the timeZero of the taskrun in which the module instance runs. The start and end of the period are included. This period overrules any relativeViewPeriods specified in the timeSeriesSets of the time series.
minNumberOfValues: The minimum required number of values in the time series to pass the check.
logLevel: Log level for the log message that is logged if a time series does not pass the check. Can be DEBUG, INFO, WARN, ERROR or FATAL. If level is error or fatal, then the module will stop running after logging the first log message.
logEventCode: Event code for the log message that is logged if a time series does not pass the check. This event code has to contain a dot, e.g. "TimeSeries.Check", because the log message is only visible to the master controller if the event code contains a dot.
logMessage: Log message that is logged if a time series does not pass the check. It is possible to use the tag %HEADER% in the logMessage. The %HEADER% tag will be replaced with the header of the time series.

Configuration Example

...

.

...

Contents of check for setting flags per time step

...

id: identifier of the check.
variableDefinition: embedded variable definition (see above).
checkRelativePeriod: The check will only consider data in this time period. This time period is relative to the timeZero of the taskrun in which the module instance runs. The start and end of the period are included. This period overrules any relativeViewPeriods specified in the timeSeriesSets of the time series.
expression: A comparison between one or more variableIds.
validatingVariableId: One or more identifiers for variables for which the flags have to be modified.
outputFlag: New flag value for time steps for which there is valid data and the expression fails. Either doubtful or unreliable.

Configuration example for checks for generating log events

Code Block

	xml
	xml

<timeSeriesAvailabilityChecker<?xml version="1.0" encoding="UTF-8"?>
<secondaryValidation xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesAvailabilityCheckersecondaryValidation.xsd">
	<timeSeries>          <variableDefinition>
		<timeSeriesId>input1<<variableId>input1</timeSeriesId>variableId>
		<timeSeriesSet>
			<moduleInstanceId>CheckImportedData<<moduleInstanceId>MinReliableValuesCheckTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>location1</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="15"/>
			<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</timeSeries>variableDefinition>
	<timeSeries><variableDefinition>
		<timeSeriesId>input2<<variableId>input2</timeSeriesId>variableId>
		<timeSeriesSet>
			<moduleInstanceId>CheckImportedData<<moduleInstanceId>MinReliableValuesCheckTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>location2</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="15"/>
			<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</timeSeries>variableDefinition>

	<minReliableValuesCheck<minNonMissingValuesCheck id="MinReliableValuesCheckMinNonMissingValuesCheck">
		<timeSeries><variable>
			<timeSeriesId>input1<<variableId>input1</timeSeriesId>variableId>
		</timeSeries>variable>
		<timeSeries><variable>
			<timeSeriesId>input2<<variableId>input2</timeSeriesId>variableId>
		</timeSeries>variable>		
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>6<<minNumberOfValues>18</minNumberOfValues>
		<logLevel>WARN<<logLevel>INFO</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough reliable values available for time series %header%</logMessage>
	</minNonMissingValuesCheck>

        <minNumberOfValuesCheck id="MinNumberOfValuesCheck">
		<variable>
			<variableId>input1</variableId>
		</variable>
		<variable>
			<variableId>input2</variableId>
		</variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>24</minNumberOfValues>
		<logLevel>DEBUG</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough values available for time series %header%</logMessage>
	</minReliableValuesCheck>
	minNumberOfValuesCheck>

        <minReliableOrDoubtfulValuesCheck id="MinReliableOrDoubtfulValuesCheck">
		<timeSeries><variable>
			<timeSeriesId>input1<<variableId>input1</timeSeriesId>variableId>
		</timeSeries>variable>
		<timeSeries><variable>
			<timeSeriesId>input2<<variableId>input2</timeSeriesId>variableId>
		</timeSeries>variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>12</minNumberOfValues>
		<logLevel>WARN</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough reliable or doubtful values available for time series %header%</logMessage>
	</minReliableOrDoubtfulValuesCheck>

	<minNonMissingValuesCheck<minReliableValuesCheck id="MinNonMissingValuesCheckMinReliableValuesCheck">
		<timeSeries><variable>
			<timeSeriesId>input1<<variableId>input1</timeSeriesId>variableId>
		</timeSeries>variable>
		<timeSeries><variable>
			<timeSeriesId>input2<<variableId>input2</timeSeriesId>variableId>
		</timeSeries>variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>18<<minNumberOfValues>6</minNumberOfValues>
		<logLevel>INFO<<logLevel>WARN</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough non-missing values available for time series %header%</logMessage>
	</minReliableValuesCheck>
</secondaryValidation>

A configuration example with for the seriesComparisonCheck is given below:

Code Block

	xml
	xml


</minNonMissingValuesCheck>
	<minNumberOfValuesCheck?xml version="1.0" encoding="UTF-8"?>
<secondaryValidation xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/secondaryValidation.xsd">

    <!-- comparison between location variable and scalar, set to unreliable -->
	<seriesComparisonCheck id="MinNumberOfValuesCheckcheckWithScalar">
		<timeSeries><variableDefinition>
			<variableId>H_obs_location1</variableId>
			<timeSeriesSet>
				<moduleInstanceId>CheckImportedData<<moduleInstanceId>SeriesComparisonCheck</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>P<parameterId>H.obs</parameterId>
				<locationSetId>locationSet1<<locationId>location1</locationSetId>locationId>
				<timeSeriesType>external historical</timeSeriesType>
				<timeStep unit="nonequidistant"minute" multiplier="15"/>
				<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
				<readWriteMode>read only</readWriteMode>			
			</timeSeriesSet>
		</timeSeries>variableDefinition>	
   		<checkRelativePeriod unit="hour" start="-12" end="0"/>	
		<minNumberOfValues>24</minNumberOfValues><expression>H_obs_location1 .ge. 13</expression>
		<logLevel>DEBUG</logLevel><validatingVariableId>H_obs_location1</validatingVariableId>	
		<logEventCode>TimeSeries.Check<<outputFlag>unreliable</logEventCode>outputFlag>
		<logMessage>Not enough values available for time series %header%</logMessage>
	</minNumberOfValuesCheck>
</timeSeriesAvailabilityChecker> </seriesComparisonCheck>
</secondaryValidation>

Page tree

Versions Compared

Old Version 11

New Version 12

Key

Secondary Validation SecondaryValidation (since 2010_01)

Contents of

checks for generating log events

Configuration Example

Contents of check for setting flags per time step

Configuration example for checks for generating log events