What is Validation Status

The progress in the validation process is called Validation Status. Besides seeing the result of a series of validation steps, the user will be able to determine the results of individual validation steps.

This functionality is available from Delft-FEWS versin 2015.02 onwards.

Use Cases

Validation Status should support the users in their specific validation practices and allow the user to confirm that the data has been adequately validated. The following use cases will be supported:

• For a time series, it will be registered per timestep, which validation steps have been executed
• The progress in the validation process  can be used as a condition for execution of a transformation/workflow
• The user gets a visual indication (tooltip, validation step columns in table view, background color and color bar) which validation steps have been performed
• The user gets extended possibilities to validate timeseries manually

Summary of Validation Status implementation

The validation procedure of an organisation can consist of various (serial) validation tasks: Primary/Secondary validation, transformation interpolation of data, visual inspection, ... In the end, a validated timeseries will have a quality flag (Reliable/Doubtful/Unreliable/Missing) as a result from the validation process. Next to the flag it is also possible to get or set information why the flag is as it is through the so-called flagSource. Since version 2012.01 FEWS stores not only the quality flags, but also the source of the flag, the so-called flagSource. So the user is able to see why a certain value is validated as unreliable, eg. due to exceeding of the hard max. The regular flagSource (which is visible in the Delft-FEWS interface as the 'Validation' item) is therefor a summary of the validation process.

 Since version 2015.02 FEWS makes it possible to configure so-called 'flagSource columns' for each validation step, in which the flagSource of the specific validation step will be recorded. From these flagSource columns the user can deduct which individual validation steps have been performed and to what result. Even when a defined validation step has been executed successfully and no quality flag has been changed, a default flagSource "OK" will be set for the respective flagSource column. The flagSource columns therefor provide details on all the validation steps. 

Functionality

FlagSource columns can be defined in a dedicated flagSourceColumns regional configuration file, see FlagSourceColumns in the Configuration guide. These are stored for each timestep for all timeseries that have been validated in the FEWS database. The following principles apply and options are available:

FlagSourceColumns are optional

When they are not defined, they will not be stored. Specific flagSource related functionality will not be available to users when the flagSourceColumns file is not configured. Also, on startup, Delft-FEWS performs a check whether all referenced flagSource ColumnId's are configured in the flagSourceColumn configuration file and throw an error/warning when there are configuration inconsistencies.

Writing of flagSources

A validation step where a flagSourceColumn has been defined will write the appropriate flagSource in case of a flag change, both to the flagSource column and the regular flagSource element. 
In case the flag is not changed, the default flagSource "OK" which indicates that the validation step has been performed, will be written to the flagSource column. In this case, the flagSource will only be written to the regular flagSource element when this is empty.
This ensures that the regular FlagSource element (visible to the user in the FEWS interface with the 'Validation' option) remains a summary of the validation status.

FlagSource columns are configured using an columnId and a storageKey

The Id of the column is only used to reference this column from the rest of the configuration. This id is not stored in the database. This id can be changed at any time without the requirement of deleting the datastore
The storageKey is an integer between 0 to 127 that is used in the datastore to reference a columnId. After changing this key you have to delete the local datastore and MC database

Order of flagSourceColumns

The order in which the flagSourceColumns are configured should reflect the sequence of the respective validation steps in the validation process.
This order is also used in the timeseriesDisplay - tableview, when the 'Validation steps' are visualized.

flagSources contained in the flagSource Columns

The flagSourceColumns can only contain the existing flagSources. Also see D Quality flags. These flagSources will automatically be set to the flagSource Column when a flag is changed in the respective validation step

To support transformation and visual validation steps, custom flagSources can be configured in the customFlagSources configuration file, see 27 CustomFlagSources.

Copying of flagSource columns to other timeseries

This is not possible with the current implementation.

Validation Status as a condition

By defining a 'visibilityControllingFlagSourceColumnId' element in the timeSeriesSet, the existence of a flagSource Column for a particular timeseries (step) can be used as a condition for a transformation. When the configured  flagSourceColumn is not existing for a particular timeseries step, the value will be perceived as 'Missing' for the transformation. All values that did not pass this validation step are removed on read. They are set to missing for equidistant timeseries and removed for non-equidistant timeseries.

Assign a FlagSource Column per module run

It is possible to add an element defaultFlagSource and a FlagSourceColumnId in a workflow, at the module activity level of the validation step. This will dictate which flagSourceColumn (FlagSourceColumnId) will be used to write the flagSources to and what flagSource will be written in case no flag is changed (defaultFlagSource).

Beschikbaar maken van het concept validatiestatus in Delft-FEWS door FlagsourceColumns te ontwikkelen die ingezet kunnen worden voor de monitoring van validatiestappen
II. Validatiestatus als conditie
De informatie in de FlagSource kolommen (validatiestappen) moet kunnen worden gebruikt als conditie voor het uitvoeren van een bepaalde module of workflow
III. Toekennen van validatiestatus per module
De FlagsourceColumn (validatiestap) moet op workflow/transformatie niveau, toegekend kunnen worden aan een tijdreeks
IV. Interface onwikkelingen Validatiestatus
A. De FlagsourceColumns zijn zichtbaar in de tijdserietabel als extra kolommen 
B. De betekenis/vertaling van een FlagsourceColumn moet geconfigureerd kunnen worden
C. De voortgang is met een (configureerbare) kleur zichtbaar in de grafiek
D. De default periode selectie (start/eindtijd van de tijdreeksen) in een validatiedisplay moet configureerbaar zijn
V. Validatiestatus Import/Export
De validatiestatus moet geimporteerd/geëxporteerd kunnen worden naar CSV/XML formaat en moet ondersteund worden in de PI Webservice
VI. Validatie via GUI
De gebruiker moet handmatig kunnen valideren (FlagsourceColumns zetten of aanpassen) voor een of meerdere validatiestappen.