26. Validation Status

Contents

Overview

Besides seeing the result of a series of validation steps, the user will be able to determine the results of individual validation steps. The progress in the validation process is called Validation Status. 

This functionality is available from Delft-FEWS version 2015.02 onwards.

For background of validation functionality and quality flags within the various Delft-FEWS displays, please see this presentation.

Use Cases

Validation Status should support the users in their specific validation practices and allow the users to confirm that the data has been adequately validated. The following use cases will be supported:

• For a time series, it will be registered per timestep, which validation steps have been performed
• The progress in the validation process  can be used to make time series data (un)available for transformations / display
• The user gets a visual indication (tooltip, validation step columns in table view, background color and color bar) which validation steps have been performed
• The user gets extended possibilities to validate timeseries manually

Summary of Validation Status implementation

The validation procedure of an organisation can consist of various (serial) validation tasks: Primary/Secondary validation, transformation interpolation of data, visual inspection, ... In the end, a validated timeseries will possibly have a quality flag (Reliable/Doubtful/Unreliable/Missing) as a result from the validation process. Since version 2012.01 FEWS stores not only the quality flags, but also the source of the flag, the so-called flagSource. So the user is able to see why a certain value is validated as unreliable, eg. due to exceeding of the hard max. The regular flagSource (which is visible in the Delft-FEWS interface as the 'Validation' item), in combination with the quality flag is therefor a summary of the validation process.

 Since version 2015.02 FEWS makes it possible to configure so-called 'flagSource columns' for each individual validation step, in which the flagSource of the specific validation step will be recorded. From these flagSource columns the user can deduct which individual validation steps have been performed and to what result. Even when a defined validation step has been executed successfully and no quality flag has been changed, a default flagSource "OK" will be set for the respective flagSource column. The flagSource columns therefor provide details on all the validation steps. 

Since version 2016.01 the concept 'Time of Validity' (Tv) has been introduced. This is defined the last timestep for which a timeseries has been validated (i.e. for which a specific flagSourceColumn has been set). Specific functionality related to the Time of Validity (Tv) becomes available for the timeseries display whenever a 'Time of Validity mode' is activated by activating a Tv button in the toolbar. The Tv is also available in de database lister.

Functionality

FlagSource columns can be defined in a dedicated flagSourceColumns regional configuration file, see FlagSourceColumns in the Configuration guide. These flagSource Columns are stored for each timestep for all timeseries that have been validated in the FEWS database. Various configuration options allow the user to harness the Validation Status in the operational process, either by visualisation, export, conditional filtering in transformations, etc.

The following principles apply and options are available:

FlagSourceColumns are optional

All Validation Status related configuration options are optional. When flagSourceColumns are not defined in the configuration, they will not be used by Delft-FEWS. When timeseries in a database contain flagSource Columns which are not configured, they will be ignored by FEWS.

Specific flagSource related functionality will not be available to users when the flagSourceColumns file is not configured.

On startup, Delft-FEWS performs a check whether all flagSource ColumnId's referenced in the configuration are defined in the flagSourceColumn configuration file. When there are configuration inconsistencies, FEWS will throw an error/warning.

Writing of flagSources

A validation step where a flagSourceColumn has been defined will write the appropriate flagSource in case of a flag change, both to the flagSource column and the regular flagSource element.

In case the flag is not changed, the default flagSource "OK" which indicates that the validation step has been performed, will be written to the flagSource column. In this case, the flagSource will only be written to the regular flagSource element when this is empty.

This ensures that the regular FlagSource element (visible to the user in the FEWS interface with the 'Validation' option) remains a summary of the validation status.

FlagSource columns are configured using an columnId and a storageKey

The Id of the column is only used to reference this column from the rest of the configuration. This id is not stored in the database. This id can be changed at any time without the requirement of deleting the datastore. 

The storageKey is an integer between 0 to 127 that is used in the datastore to reference a columnId. After changing this key you have to delete the local datastore and MC database.

flagSources contained in the flagSource Columns

The flagSourceColumns can only contain the existing flagSources. Also see D Time Series Flags. These flagSources will automatically be set to the flagSource Column when a flag is changed in the respective validation step

To support transformation and visual validation steps, custom flagSources can be configured in the customFlagSources configuration file, see 27 CustomFlagSources.

Validation Status as a condition

By defining a 'visibilityControllingFlagSourceColumnId' element in the timeSeriesSet, the existence of a flagSource Column for a particular timeseries (step) can be used as a condition for for example a transformation. When the configured  flagSourceColumn is not existing for a particular timeseries step, the value will be perceived as 'Missing' for the transformation. All values that did not pass this validation step are removed on read. They are set to missing for equidistant timeseries and removed for non-equidistant timeseries.

This element can also be used to prevent non-validated data to be exportable through a PI-Webservice. If the timeseriesSets in the filterId that is used for the Webservice are configured with a visibilityControllingFlagSourceColumnId, timesteps that do not have the correct validation status will not be exported.

Assign a FlagSource Column per module run

It is possible to add an element defaultFlagSource and a FlagSourceColumnId in a workflow, at the module activity level of the validation step. This will dictate which flagSourceColumn (FlagSourceColumnId) will be used to write the flagSources to and what flagSource will be written in case no flag is changed (defaultFlagSource).

The interpolation serial transformation also has the possibility to assign a custom flagSource to a flagSourceColumn for interpolated values.

Example

<transformation id="interpolation serial linear">
	<interpolationSerial>
		<linear>
			<inputVariable>
				<variableId>input</variableId>
			</inputVariable>
			<maxGapLength>5</maxGapLength>
			<outputVariable>
				<variableId>output</variableId>
			</outputVariable>
			<outputComment>outputCommentTest</outputComment>
            <outputCustomFlagSourceId>INTP</outputCustomFlagSourceId>
		</linear>
	</interpolationSerial>
</transformation>

Validation Status in the GUI (Time series diplay)

An option 'Validation Steps' is available in the table view options (when flagSource Columns are configured). When this option is activated, all flagSource Columns that are either filled (the validation step has been performed) or are configured 'editable' will be shown next to the value column. The header of the flagSource Columns is equal to the configured 'shortName' and when the user hoovers above the header, a configured tooltip for the flagSource Column is shown.

The order in which the flagSourceColumns are configured should reflect the sequence of the respective validation steps in the validation process. This order is also used in the timeseriesDisplay - tableview, when the 'Validation Steps' are visualized. Note that the order in the configuration file is from top (first validation step) to bottom (last validation step), in the interface the order is from right (first) to left (last). 

Mark and edit Validation Status

The regular 'Validation' column cell for each individual timeseries will get background color that is equal to the configured color of the last validation step that has been performed for that timestep. When a distinctive color is used for a specific validation step that is of interest to the user, it is easy to gauge  the progress of validation from the color. This color is also visible as a bar chart under the timeseries graph, and represents the last validation step that has been executed for all timeseries.

It is possible to include an explicit manual validation step in the validation procedure using the 'Mark' option that is available in the toolbar when the timeseries display is in edit mode and flagSources have been configured. This allows the user to set a flag (or not) for selected timeseries and timesteps and at the same time indicate for what flagSource Column (e.g. 'Visual_Inspection') which flagSource (e.g. 'OK') should be written.  

Validation History

When a data value is deleted (either by the user or by a different action), the flagSources for that timestep will remain. For an individual value in the time series table, the history of that value can be shown using the right-mouse menu - Show history (CTRL+H). This menu also shows the 'state' of the flagSourceColumns for each time the value was changed in the datastore.

Time of Validity

Since 2016.01 it is possible to open up 'Time of Validity' (Tv) related functionality in the timeSeries display, by configuring a timeOfValidity element in the flagSourceColumns.xml configuration file. 

Activating Time of Validity 'mode'

A Tv button becomes available in the time series button panel, which activates all 'Time of Validity' related functionality. The button is only be visible in the button panel, when the timeOfValidity element has been defined in the flagSourceColumns.xml file. Note that none of the Tv functionality is available when the Tv button is not activated.

Besides the functionality as described below, when the Tv button from the toolbar is activated, the time series display will be set to 'edit mode' automatically.

Definition Time of Validity

The time of Validity is defined as the last timestep for which a timeseries has been validated with a particular validation step (in other words: for which a specific flagSourceColumn contains a flagSource). The Time of Validity of a single timeseries in a display is calculated according to the following logic:

• the TimeOfValidity flagSourceColumn, as defined in flagSourceColumns.xml, will be used
• The timeseries is read for the maximum relative view period timeOfValiditySearchPeriod, as defined globally in the timeseriesDisplayConfig.xml configuration file
• The timestep before the first timestep where the Tv flagSource is not available is considered as the Tv for that time series
• When no single timestep within the timeOfValiditySearchPeriod has the Tv flagSourceColumn filled, the timeseries will have no Tv
• When the first timestep within the timeOfValiditySearchPeriod does not have the flagSourceColumn filled but later timesteps do have the Tv flagSource, the Time of Validity will be equal to this first timestep within the timeOfValiditySearchPeriod
  
  

An example for a single timeseries is given in the image below

The Tv (Time of Validity) for a display node needs to be calculated by aggregating the Tv's for all the timeseries displayed under that node. Display nodes are configured in the displayGroups.xml configuration file.

• The Tv furthest in the past of all the time series under the display node will be considered as the display Time of Validity.
• The Tv also needs to be calaculated for the (sub)directories in which the displaynodes are contained.
• Read Only time series should be ignored when calculating Time of Validity
• Recalculating (and displaying the (updated)) Tv should take place after every save action.

Time of Validity in Database Viewer

When the timeOfValidity element has been defined in the flagSourceColumns.xml file, the Time of Validity will be calculated for  each timeseries in the database lister. Note that - different from the time series in the time series display - the Tv is calculated based on the complete time series.

Time of Validity marker

When Time of Validity has been configured, a Tv marker for the time series display will be displayed  in the timeSeries display (when configured).

Tv in displayNode text

The Tv that is calaculated for all timeseries in a display node will be shown in the display node text.

Format:
<displayGroupName> <time>

The position of the Tv time will be either before or after the displayGroupName and can be toggled through the right mouse menu ('Time of Validity position'). The dateTimeFormat as defined in the Explorer.xml configuration file is used, where the :ss (seconds) is removed. The color of the time will be the same as the Tv marker (and black by default, when the Tv marker is not configured).

Recalculation and displaying the (updated) Tv  takes place after every save action.

Ordering of display nodes based on Tv 

When the Tv mode is activated, the displaygroups nodes are dynamically ordered based on the Tv of that display node

• The node with the Tv furthest in the past will be displayed on top within the (sub)directory.
• When Tv's are equal, the order will be as configured in the displayGroup.xml file
• (Sub)directory display nodes will show the Tv as an aggregation of the nodes/directories in that (sub)directory

It is noted that the this automatic ordering of the display nodes within a (sub)directory will only be executed at the moment the Tv button is selected. When a validator wants to update the sorting, the Tv button can be de-activated and activated again.

View period and cursor behaviour

A required default view period for the time of Validity (Tv) has be configured to define the time span of data displayed in the time series display whenever the Tv button is selected. The view period is relative to the Tv of the time series in the display and not to T0. This view period is enforced every time the user switches to another display node. 

Also, when Tv button is selected, the cursor will default to the last timestep of the view period, whenever:

• the user selects switches from display node
• the user changes the view period (use case is CTRL+ALT+Right quickkey to change viewperiod, but this also works for manual zooming)

Note: the cursor will update to the last time step for which data is available

Update Tv flagSource column

When the Tv button is selected, the user is able to update the flagSource column related to Tv for all timesteps up to the current position of the cursor (current selected timestep) with the defaultFlagSource as defined in the flagSourceColumn configuration file.

• using an option through the rightmouse menu
• using a simple keyboard shortcut: CTRL+SHIFT+T
  
  

The update will only take place for the view period of the display. When this update function is used when Tv is outside the view period of the display, a pop up will be displayed warning the user that a validation gap would be created. The update action will not be performed in this case. This might for example happen when the validator updates the flagSourceColumns using the functionality described above - without saving. Delft FEWS does not know in that case that the Tv should be updated, as the Time of Validity is only recalculated after the timeSeries have been saved.

Export of flagSourceColumns

In the FEWS client, it is possible to use the validation status as a condition for the interactive export. A specific flagSource Column can be selected which is used as a filter for the export. The flagSource Columns are included for the PI-XML data type and will always be exported when available. The GeneralCSV datatype can be configured as an interactive export type as well. The flagSource columns that should be exported can be configured in the explorer.xml configuration file.

In the regular timeseries exports, the export of flagSourceColumns is supported for the PI-XML format and the GeneralCSV format. In PI-XML, the flagSource columns will always be exported. When exporting using the generalCsv export format, inclusion of flagSourceColumns can be defined in the configuration with the use of Table Layout.

Import of flagSourceColumns

The flagSource Columns are included for the PI-XML data type and will be imported.

The GeneralCSV datatype can be configured as an import type as well. The flagSource columns that should be imported can be configured in the explorer.xml configuration file.

Below is given an example on how to configure a generalCSV import with flagSource columns.

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
	<import>
		<general>
			<importType>generalCSV</importType>
			<folder>../junit_test_output/nl/wldelft/fews/system/plugin/dataImport/TimeSeriesImportTestData/import/generalcsv</folder>
			<table>
				<dateTimeColumn name="DATE" pattern="dd-MM-yy HH:mm"/>
				<locationColumn name="LOC"/>
				<unitColumn name="UNIT"/>
				<parameterColumn name="PARAM"/>
				<flagSourceColumn id="A" name="FS_A"/>
				<!-- column with name fs a will be mapped to the flag source column with identifier A -->
				<flagSourceColumn id="B" name="FS_B"/>
				<flagSourceColumn id="C" name="FS_C"/>
				<flagSourceColumn id="D" name="FS_D"/>
				<flagSourceColumn id="E" name="FS_E"/>
				<valueColumn name="VALUE"/>
			</table>
			<logWarningsForUnmappableLocations>true</logWarningsForUnmappableLocations>
			<logWarningsForUnmappableParameters>true</logWarningsForUnmappableParameters>
			<logWarningsForUnmappableQualifiers>true</logWarningsForUnmappableQualifiers>
			<maxLogWarnings>1000</maxLogWarnings>
			<missingValue>-999</missingValue>
			<importTimeZone>
				<timeZoneOffset>+01:00</timeZoneOffset>
			</importTimeZone>
			<dataFeedId>generalCSV</dataFeedId>
		</general>
	</import>
</timeSeriesImportRun>

DATE,LOC,VALUE,UNIT,PARAM,FS_A,FS_B,FS_C,FS_D,FS_E
01-01-81 00:00,H-2001,2.3,m,P.m,OK,OK,OK,OK,OK

Copying of flagSource columns to other timeseries

It is possible to manually copy and paste the flagSources in the flagSource Columns to other flagSource Columns (from the same, or from other timeseries), when in edit mode.

It is not possible to copy flagSource columns in a transformation to another timeseries.