Data quality flags

Quality flags

Quality flags describe the quality and origin of data.
The Flags available in Delft-FEWS are a based on a combination of two components: the origin and the quality of the data.

Origin of the data: Original / Completed / Corrected
Quality of the data: Reliable / Doubtful / Unreliable

Unreliable Data will not be used for calculations like transformations or external models.
It can still be viewed and will be exported with a regular exportModule, as long as the unreliable flag is shown with it.
No difference is made between historic and forecast data when it comes to Quality Flags.
In the data model of Delft-FEWS this difference is captured in the time series type definition.
External sources may either be an actual external source, a forecasting module or a transformation.

Additional Quality flags

In addition, there are a few other quality flags (or more broadly, time series flags) defined in Delft-FEWS, these are listed and described in the table below.

You can define custom quality flags. This can be useful to store external quality flag information that can't be mapped to the flags known to Delft-FEWS.
Custom Flags have no meaning, other than to store this extra bit of metadata. The <name> and <description> are not used anywhere in Delft-FEWS (e.g. in the GUI) and only serve as additional information to the Flag mappings defined in the file itself.

ValidationRuleSet labels

The legend of the quality labels also includes optional labels that can be configured in validationRuleSet.xml.
These labels are unrelated to the QualityFlags described on this page.

Enumeration of quality flags

This leads to the following quality flags available in Delft-FEWS.

Table 1 Enumeration of quality flags

Enumeration	Description
0	Original/Reliable The data value is the original value retrieved from an external source and it successfully passes all validation criteria set.
1	Corrected/Reliable The original value was removed and corrected. Correction may be through interpolation or manual editing.
2	Completed/Reliable Original value was missing. Value has been filled in through interpolation, transformation (e.g. stage discharge) or a model.
3	Original/Doubtful Observed value retrieved from external data source. Value is valid, but marked as suspect due to soft validation limits being exceeded.
4	Corrected/Doubtful The original value was removed and corrected. However, the corrected value is doubtful due to validation limits.
5	Completed/Doubtful Original value was missing. Value has been filled in as above, but resulting value is doubtful due to limits in transformation/interpolation or input value used for transformation being doubtful.
6	Original/Unreliable Observed value retrieved from external data source. Value is invalid due to validation limits set. Value is marked as unreliable and will not be used in transformations and model calculations.
7	Corrected/Unreliable The original value was removed and corrected. However, corrected value is unreliable
8	Completed/Unreliable Original value was missing. Value has been filled in as above, but resulting value is unreliable,
9	Missing Missing value in originally observed series. Note: this is a special form of both Original/Unreliable and Original/Reliable.
10	Deleted Only used for non-equidistant series when timestep is deleted
11, 12, 13	Not used
14	Accumulation Reset Used for rainfall data processing when the storage is reset (canister reset), see https://publicwiki.deltares.nl/spaces/FEWSDOC/pages/120819119/Precipitation

FlagSource

Next to the flag it is also possible to get or set information why the flag is as it is through the so-called flagSource.
Delft-FEWS stores not only the quality flags, but also the source of the flag, the so-called flagSource (Since version 2012.01).
The user is able to see why a certain value is validated as unreliable, eg. due to exceeding of the hard max.

Table 1 List of Flag Sources

flagSource	Description
CA	Conditional aggregation
FC	secondary validation, flag comparison
HN	hard min
HX	hard max
IMP	flag is imported
MAN	manual edit
MK	secondary validation, Mann-Kendall test
OSC	oscillation
ROF	rate of fall
ROR	rate of rise
SC	secondary validation, series comparison
SFP	start flag persistency
SN	soft min
SR	same reading
SVP	secondary validation, Flag persistency (FlagPersistencyCheck)
SX	soft max
TS	temporary shift

Page tree