...

What is Validation Status

The progress in the validation process is called Validation Status. Besides seeing the result of a series of validation steps, the user will be able to determine the results of individual validation steps. The progress in the validation process is called Validation Status. 

This functionality is available from Delft-FEWS versin version 2015.02 onwards.

Use Cases

Validation Status should support the users in their specific validation practices and allow the user users to confirm that the data has been adequately validated. The following use cases will be supported:

• For a time series, it will be registered per timestep, which validation steps have been executedperformed
• The progress in the validation process  can be used as a condition for execution of a transformation/workflow
• The user gets a visual indication (tooltip, validation step columns in table view, background color and color bar) which validation steps have been performed
• The user gets extended possibilities to validate timeseries manually

...

The validation procedure of an organisation can consist of various (serial) validation tasks: Primary/Secondary validation, transformation interpolation of data, visual inspection, ... In the end, a validated timeseries will possibly have a quality flag (Reliable/Doubtful/Unreliable/Missing) as a result from the validation process. Next to the flag it is also possible to get or set information why the flag is as it is through the so-called flagSource. Since version 2012.01 FEWS stores not only the quality flags, but also the source of the flag, the so-called flagSource. So the user is able to see why a certain value is validated as unreliable, eg. due to exceeding of the hard max. The regular flagSource (which is visible in the Delft-FEWS interface as the 'Validation' item), in combination with the quality flag is therefor a summary of the validation process.

 Since version 2015.02 FEWS makes it possible to configure so-called 'flagSource columns' for each individual validation step, in which the flagSource of flagSource of the specific validation step will be recorded. From these flagSource columns the user can deduct which individual validation steps have been performed and to what result. Even when a defined validation step has been executed successfully and no quality flag has been changed, a default flagSource "OK" will be set for the respective flagSource column. The flagSource columns therefor provide details on all the validation steps. 

...

FlagSource columns can be defined in a dedicated flagSourceColumns regional configuration file, see FlagSourceColumns in the Configuration guide. These flagSource Columns are stored for each timestep for all timeseries that have been validated in the FEWS database.  The Various configuration options allow the user to harness the Validation Status in the operational process, either by visualisation, export, conditional filtering in transformations, etc.

The following principles apply and options are available:

FlagSourceColumns are optional

When they All Validation Status related configuration options are optional. When flagSourceColumns are not defined in the configuration, they will not be used by Delft-FEWS. When timeseries in a database contain flagSource Columns which are not configured, they will be ignored by FEWS.

stored. Specific flagSource related functionality will not be available to users when the flagSourceColumns file is not configured. Also, on

On startup, Delft-FEWS performs a check whether all referenced flagSource ColumnId's referenced in the configuration are configured defined in the flagSourceColumn configuration file and throw . When there are configuration inconsistencies, FEWS will throw an error/warning when there are configuration inconsistencies.

Writing of flagSources

A validation step where a flagSourceColumn has been defined will write the appropriate flagSource in case of a flag change, both to the flagSource column and the regular flagSource element. 

In case the flag is not changed, the default flagSource "OK" which indicates that the validation step has been performed, will be written to the flagSource column. In this case, the flagSource will only be written to the regular flagSource element when this is empty.

This ensures that the regular FlagSource element (visible to the user in the FEWS interface with the 'Validation' option) remains a summary of the validation status.

...

The Id of the column is only used to reference this column from the rest of the configuration. This id is not stored in the database. This id can be changed at any time without the requirement of deleting the datastore. 

The storageKey is an integer between 0 to 127 that is used in the datastore to reference a columnId. After changing this key you have to delete the local datastore and MC database.

flagSources contained in the flagSource Columns

...