Contents of checks for counting reliable, doubtful, unreliable and missing values

The minNumberOfValuesCheck, minNonMissingValuesCheck, minReliableOrDoubtfulValuesCheck and minReliableValuesCheck all consist of the following elements:

  • id: Identifier of the check. This is only used in log messages and exception messages.
  • variable: One or more time series that need to be checked. This can be either an embedded timeSeriesSet or a reference to a variableDefinition defined at the start of the configuration file. If this contains multiple time series (e.g. for multiple locations), then each time series is checked individually.
  • checkRelativePeriod: The check will only consider data in this time period. This time period is relative to the timeZero of the taskrun in which the module instance runs. The start and end of the period are included. This period overrules any relativeViewPeriods specified in the timeSeriesSets of the time series.
  • minNumberOfValues: The minimum required number of values in the time series to pass the check.
  • logLevel: Log level for the log message that is logged if a time series does not pass the check. Can be DEBUG, INFO, WARN, ERROR or FATAL. If level is error or fatal, then the module will stop running after logging the first log message.
  • logEventCode: Event code for the log message that is logged if a time series does not pass the check. This event code has to contain a dot, e.g. "TimeSeries.Check", because the log message is only visible to the master controller if the event code contains a dot.
  • logMessage: Log message that is logged if a time series does not pass this check. It is possible to use the following tags: %HEADER% and %LOCATION_NAME%. The %HEADER% tag will be replaced with the header of the time series. The %LOCATION_NAME% tag will be replaced with the name of the location of the time series. For minNumberOfValuesCheck, minNonMissingValuesCheck, minReliableOrDoubtfulValuesCheck and minReliableValuesCheck it is also possible to use additional tags between "@" signs, e.g. "@ID@", that refer to location attributes that are defined in the locationSets config file. The values of the location attributes can be different for different locations. The location attributes are evaluated first, then the other tags (between % signs) are evaluated. See below, the complete list.
  • onErrorResumeNext When true, makes the secondary validation continue when an error logging is applied.


One can use the next tags in the logmessage:

  • %HEADER%
  • %LOCATION_NAME%
  • %LOCATION_ID% displays the locationId where the alterations took place
  • any location attribute by using @ signs
  • %AMOUNT_CHANGED_FLAGS%
  • %CHECK_ID% puts in the id of the check that caused the flags to be altered.
  • %OUTPUT_FLAG% contains the flag that has been set
  • %EXPRESSION%, which is the expression that caused the flags to be altered. 
  • %PARAMETER_ID%  contains  the parameterId where the alterations took place.
  • %PARAMETER_NAME% contains the name of the parameter where the alterations took place.
  • %PERIOD% contains the period in which flags were changed.
  • %NONE% hides the autogenerated location and parameter in the log message.
  • %VALUE%, which will be replaced with the latest value for the validating timeseries (for SeriesComparisonCheck)

Configuration example for checks on amounts of reliable, doubtful, unreliable and missing values

<?xml version="1.0" encoding="UTF-8"?>
<secondaryValidation xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/secondaryValidation.xsd">
          <variableDefinition>
		<variableId>input1</variableId>
		<timeSeriesSet>
			<moduleInstanceId>MinReliableValuesCheckTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>location1</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="15"/>
			<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</variableDefinition>
	<variableDefinition>
		<variableId>input2</variableId>
		<timeSeriesSet>
			<moduleInstanceId>MinReliableValuesCheckTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>location2</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="15"/>
			<!-- any relativeViewPeriod here will always be overruled by checkRelativePeriod in each check -->
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</variableDefinition>

	<minNonMissingValuesCheck id="MinNonMissingValuesCheck">
		<variable>
			<variableId>input1</variableId>
		</variable>
		<variable>
			<variableId>input2</variableId>
		</variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>18</minNumberOfValues>
		<logLevel>INFO</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough values available for time series %header%, district @DISTRICT@</logMessage>
	</minNonMissingValuesCheck>

        <minNumberOfValuesCheck id="MinNumberOfValuesCheck">
		<variable>
			<variableId>input1</variableId>
		</variable>
		<variable>
			<variableId>input2</variableId>
		</variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>24</minNumberOfValues>
		<logLevel>DEBUG</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough values available for time series %header%, district @DISTRICT@</logMessage>
	</minNumberOfValuesCheck>

        <minReliableOrDoubtfulValuesCheck id="MinReliableOrDoubtfulValuesCheck">
		<variable>
			<variableId>input1</variableId>
		</variable>
		<variable>
			<variableId>input2</variableId>
		</variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>12</minNumberOfValues>
		<logLevel>WARN</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough values available for time series %header%, district @DISTRICT@</logMessage>
	</minReliableOrDoubtfulValuesCheck>

	<minReliableValuesCheck id="MinReliableValuesCheck">
		<variable>
			<variableId>input1</variableId>
		</variable>
		<variable>
			<variableId>input2</variableId>
		</variable>
		<checkRelativePeriod unit="hour" start="-12" end="0"/>
		<minNumberOfValues>6</minNumberOfValues>
		<logLevel>WARN</logLevel>
		<logEventCode>TimeSeries.Check</logEventCode>
		<logMessage>Not enough values available for time series %header%, district @DISTRICT@</logMessage>
	</minReliableValuesCheck>
</secondaryValidation>

Note that in this example the location attribute DISTRICT is defined and used in the log message.

  • No labels

Disclaimer