26. Validation Status

Contents

Overview

Besides seeing the result of a series of validation steps, the user will be able to determine the results of individual validation steps. The progress in the validation process is called Validation Status. 

This functionality is available from Delft-FEWS version 2015.02 onwards.

Use Cases

Validation Status should support the users in their specific validation practices and allow the users to confirm that the data has been adequately validated. The following use cases will be supported:

• For a time series, it will be registered per timestep, which validation steps have been performed
• The progress in the validation process  can be used as a condition for execution of a transformation/workflow
• The user gets a visual indication (tooltip, validation step columns in table view, background color and color bar) which validation steps have been performed
• The user gets extended possibilities to validate timeseries manually

Summary of Validation Status implementation

The validation procedure of an organisation can consist of various (serial) validation tasks: Primary/Secondary validation, transformation interpolation of data, visual inspection, ... In the end, a validated timeseries will possibly have a quality flag (Reliable/Doubtful/Unreliable/Missing) as a result from the validation process. Since version 2012.01 FEWS stores not only the quality flags, but also the source of the flag, the so-called flagSource. So the user is able to see why a certain value is validated as unreliable, eg. due to exceeding of the hard max. The regular flagSource (which is visible in the Delft-FEWS interface as the 'Validation' item), in combination with the quality flag is therefor a summary of the validation process.

 Since version 2015.02 FEWS makes it possible to configure so-called 'flagSource columns' for each individual validation step, in which the flagSource of the specific validation step will be recorded. From these flagSource columns the user can deduct which individual validation steps have been performed and to what result. Even when a defined validation step has been executed successfully and no quality flag has been changed, a default flagSource "OK" will be set for the respective flagSource column. The flagSource columns therefor provide details on all the validation steps. 

Functionality

FlagSource columns can be defined in a dedicated flagSourceColumns regional configuration file, see FlagSourceColumns in the Configuration guide. These flagSource Columns are stored for each timestep for all timeseries that have been validated in the FEWS database. Various configuration options allow the user to harness the Validation Status in the operational process, either by visualisation, export, conditional filtering in transformations, etc.

The following principles apply and options are available:

FlagSourceColumns are optional

All Validation Status related configuration options are optional. When flagSourceColumns are not defined in the configuration, they will not be used by Delft-FEWS. When timeseries in a database contain flagSource Columns which are not configured, they will be ignored by FEWS.

Specific flagSource related functionality will not be available to users when the flagSourceColumns file is not configured.

On startup, Delft-FEWS performs a check whether all flagSource ColumnId's referenced in the configuration are defined in the flagSourceColumn configuration file. When there are configuration inconsistencies, FEWS will throw an error/warning.

Writing of flagSources

A validation step where a flagSourceColumn has been defined will write the appropriate flagSource in case of a flag change, both to the flagSource column and the regular flagSource element.

In case the flag is not changed, the default flagSource "OK" which indicates that the validation step has been performed, will be written to the flagSource column. In this case, the flagSource will only be written to the regular flagSource element when this is empty.

This ensures that the regular FlagSource element (visible to the user in the FEWS interface with the 'Validation' option) remains a summary of the validation status.

FlagSource columns are configured using an columnId and a storageKey

The Id of the column is only used to reference this column from the rest of the configuration. This id is not stored in the database. This id can be changed at any time without the requirement of deleting the datastore. 

The storageKey is an integer between 0 to 127 that is used in the datastore to reference a columnId. After changing this key you have to delete the local datastore and MC database.

flagSources contained in the flagSource Columns

The flagSourceColumns can only contain the existing flagSources. Also see D Quality flags. These flagSources will automatically be set to the flagSource Column when a flag is changed in the respective validation step

To support transformation and visual validation steps, custom flagSources can be configured in the customFlagSources configuration file, see 27 CustomFlagSources.

Copying of flagSource columns to other timeseries

This is not possible with the current implementation.

Validation Status as a condition

By defining a 'visibilityControllingFlagSourceColumnId' element in the timeSeriesSet, the existence of a flagSource Column for a particular timeseries (step) can be used as a condition for for example a transformation. When the configured  flagSourceColumn is not existing for a particular timeseries step, the value will be perceived as 'Missing' for the transformation. All values that did not pass this validation step are removed on read. They are set to missing for equidistant timeseries and removed for non-equidistant timeseries.

Assign a FlagSource Column per module run

It is possible to add an element defaultFlagSource and a FlagSourceColumnId in a workflow, at the module activity level of the validation step. This will dictate which flagSourceColumn (FlagSourceColumnId) will be used to write the flagSources to and what flagSource will be written in case no flag is changed (defaultFlagSource).

Validation Status in the GUI

An option 'Validation Steps' is available in the table view options (when flagSource Columns are configured). When this option is activated, all flagSource Columns that are either filled (the validation step has been performed) or are configured 'editable' will be shown next to the value column. The header of the flagSource Columns is equal to the configured 'shortName' and when the user hoovers above the header, a configured tooltip for the flagSource Column is shown.

The order in which the flagSourceColumns are configured should reflect the sequence of the respective validation steps in the validation process. This order is also used in the timeseriesDisplay - tableview, when the 'Validation Steps' are visualized.

The regular 'Validation' column cell for each individual timeseries will get background color that is equal to the configured color of the last validation step that has been performed for that timestep. When a distinctive color is used for a specific validation step that is of interest to the user, it is easy to gauge  the progress of validation from the color. This color is also visible as a bar chart under the timeseries graph, and represents the last validation step that has been executed for all timeseries.

It is possible to include an explicit manual validation step in the validation procedure using the 'Mark' option that is available in the toolbar when the timeseries display is in edit mode and flagSources have been configured. This allows the user to set a flag (or not) for selected timeseries and timesteps and at the same time indicate for what flagSource Column (e.g. 'Visual_Inspection') which flagSource (e.g. 'OK') should be written.  

Export of flagSourceColumns

In the FEWS client, it is possible to use the validation status as a condition for the interactive export. A specific flagSource Column can be selected which is used as a filter for the export. In the export, the flagSource Columns are included for the PI-XML and GeneralCSV export types, when the user indicates that validation/comments should be part of the export