...
What | nameofinstance.xml |
|---|---|
Description | Configuration for the new version of the transformation module |
schema location |
Entry in ModuleDescriptors
<className>nl.wldelft.fews.system.plugin.transformationmodule.TransformationModule</className>
</moduleDescriptor>
Contents
| Table of Contents |
|---|
| Children Display |
|---|
...
TransformHBV_Inputs 1.00 default.xml.
TransformHBV_Inputs | File name for the TransformHBV_Inputs configuration. |
1.00 | Version number |
default | Flag to indicate the version is the default configuration (otherwise omitted). |
The configuration for the transformation module consists of two parts: transformation configuration files in the Config/ModuleConfigFiles directory and coefficient set configuration files in the Config/CoefficientSetsFiles directory.
...
The validation rules are optional in the configuration and can be used to define the outputflag and the custom flagsource of the output value based on the number of missing values/unreliables values and/or the number of doubtful values in the used input values per aggregation timestep. The available output flags are reliable, doubtful and missing.
...
Below the configuration of the basic example which was described above.
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputMissingPercentage>10</inputMissingPercentage> <outputValueFlag>reliable</outputValueFlag> </validationRule> <validationRule> <inputMissingPercentage>100</inputMissingPercentage> <outputValueFlag>missing</outputValueFlag> </validationRule> |
...
Below shows a configuration example in which the rules above are implemented.
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputMissingPercentage>15</inputMissingPercentage> <outputValueFlag>reliable</outputValueFlag> </validationRule> <validationRule> <inputMissingPercentage>40</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> </validationRule> <validationRule> <inputMissingPercentage>100</inputMissingPercentage> <outputValueFlag>missing</outputValueFlag> </validationRule> |
...
Below an example in which the output is reliable when there are no missing values in the input and when the percentage if missing values is less than 15%. However in the first case the output doesn't get a custom flagsource assigned while in the second case the output gets a custom flagsource assigned which is visible in the GUI to indicate that a output value was calculated but that missing values were found in the input.
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputMissingPercentage>0</inputMissingPercentage> <outputValueFlag>reliable</outputValueFlag> </validationRule> <validationRule> <inputMissingPercentage>15</inputMissingPercentage> <outputValueFlag>reliable</outputValueFlag> <outputCustomFlagSourceId>CA</outputCustomFlagSourceId> </validationRule> <validationRule> <inputMissingPercentage>40</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> </validationRule> <validationRule> <inputMissingPercentage>100</inputMissingPercentage> <outputValueFlag>missing</outputValueFlag> </validationRule> |
...
Below a configuration example
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputDoubtfulPercentage>10</inputDoubtfulPercentage> <inputMissingPercentage>0</inputMissingPercentage> <outputValueFlag>reliable</outputValueFlag> </validationRule> <validationRule> <inputDoubtfulPercentage>30</inputDoubtfulPercentage> <inputMissingPercentage>0</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> <outputCustomFlagSourceId>D1</outputCustomFlagSourceId> </validationRule> <validationRule> <inputDoubtfulPercentage>60</inputDoubtfulPercentage> <inputMissingPercentage>0</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> <outputCustomFlagSourceId>D2</outputCustomFlagSourceId> </validationRule> <validationRule> <inputDoubtfulPercentage>100</inputDoubtfulPercentage> <inputMissingPercentage>0</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> <outputCustomFlagSourceId>D3</outputCustomFlagSourceId> </validationRule> <validationRule> <inputMissingPercentage>15</inputMissingPercentage> <outputValueFlag>reliable</outputValueFlag> <outputCustomFlagSourceId>CA</outputCustomFlagSourceId> </validationRule> <validationRule> <inputMissingPercentage>40</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> </validationRule> <validationRule> <inputMissingPercentage>100</inputMissingPercentage> <outputValueFlag>missing</outputValueFlag> </validationRule> |
...
In the examples above the inputMissingValuePercentage and the inputDoubtfulPercentage was configured hard-coded in the configuration file. However it is also possible to make a reference to an attribute of a location. To reference to an attribute the referenced attribute should be placed within @.
| Code Block | ||
|---|---|---|
| ||
<inputMissingPercentage>@MV@</inputMissingPercentage> |
...
To explain the concept of the validation rules more the table below shows the input time series and the output time series of an aggregation accumulative tranformation which uses the validation rules which are shown above in the last eexample.
Time | Input value | input flag | Output value | output flag | custom flagsource |
|---|---|---|---|---|---|
1-1-2012 00:15 |
1-1-2012 00:30 | 1 | ||||
1 |
1-1-2012 00:30
1
-1-2012 00:45 | 1 |
1-1-2012 01:00 | 1 |
3 | doubtful | - | |
1-1-2012 01:15 |
1-1-2012 01:30 | 1 |
1-1-2012 01:45 |
1-1-2012 02:00 | 1 |
NaN | - | - | ||
1-1-2012 02:15 | 1 |
1-1-2012 02:30 | 1 | doubtful |
1-1-2012 02:45 | 1 |
1-1-2012 03:00 | 1 |
4 | doubtful | D1 | ||
1-1-2012 03:15 | 1 |
1-1-2012 03:30 | 1 |
1-1-2012 03:45 | 1 |
1-1-2012 04:00 | 1 |
4 | reliable |
...
The first output value is set to doubtful. Because in this case the total percentage of missing values is 25%. Which means that the following rule is applied.
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputMissingPercentage>40</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> </validationRule> |
The second output value is a missing value because in this case the percentage of missing values is equal to 50%. This means that in this case the following rule will be appplied.
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputMissingPercentage>100</inputMissingPercentage> <outputValueFlag>missing</outputValueFlag> </validationRule> |
...
The third output value is set to doubtful. The input doesn't contain missing values but has a single doubtful input value. The percentage of doubtful values in the input is therefore 25% which means that the following rule will be applied.
| Code Block | ||
|---|---|---|
| ||
<validationRule> <inputDoubtfulPercentage>30</inputDoubtfulPercentage> <inputMissingPercentage>0</inputMissingPercentage> <outputValueFlag>doubtful</outputValueFlag> <outputCustomFlagSourceId>D1</outputCustomFlagSourceId> </validationRule> |
...
Since FEWS 2017.02 it is possible to configure if manual edits should be preserved. This setting applies to all transformations that are configured. The default is false. For an example configuration see:
| Code Block | ||
|---|---|---|
| ||
<transformationModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/transformationModule.xsd" version="1.0">
<preserveManualEdits>true</preserveManualEdits> |
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
<transformation id="merge"> <merge> <simple> <inputVariable> <variableId>Wiski</variableId> </inputVariable> <inputVariable> <variableId>Server</variableId> </inputVariable> <fillGapConstant>0</fillGapConstant> <outputVariable> <variableId>merge1</variableId> </outputVariable> </simple> </merge> <description>transformation description</description> </transformation></outputVariable> </simple> </merge> <description>transformation description</description> </transformation> |
Copying comments, flags and flag sources from input to output
<TODO>
Preventing previously calculated values to be overwritten with missings
Delft-FEWS processes data in moving windows compared to the timezero of the workflow. These workflows can be run several times per day. In certain conditions this could lead a transformation to calculate a missing value for a datetime for which earlier a correct value was calculated. This can lead to warnings such as:
Existing value overwritten by missing
Suspicious write action. Long time series written with only changes at the start and at the end. If this happens often this will explode the database.
This mechanism is illustrated with the image below, which shows a water level for which an average per day is calculated. The box shows the moving window (relativeViewPeriod). The image shows the daily values (red dots) calculated in the first run.
In a later run (illustrated below), there are not sufficient values to calculate a daily value for 24 December. So this run will return a missing value for 24 December (while calculating a new value for 27 December.
If these transformations would directly write to the same output timeserie (in this case with timeseriesType “external historical”), the later run would cause the average water level for December 24 to be overwritten with a missing value. The log would include warnings about this!
To prevent this, the output timeserie should be of timeserieType “temporary”, and next be merged (merge / simple transformation) with the final timeserie. This way, missing values will not overwrite previously calculated values.
Run transformations for a set of selected locations
...
It is possible to run a workflow for a selected set of locations from the IFD, the task dialog and the manual forecast dialog. By default workflows cannot be run for a selected set of locations. To enable this the option allowSelection should be set to true in the workflowdescriptor of the workflow. Below an example.
| Code Block | ||
|---|---|---|
| ||
<workflowDescriptor id="FillRelations" forecast="false" visible="true"autoApprove="false"> <description>Met deze taak worden de gaten groter dan 2 uur gevuld dmv. relaties.</description> <allowSelection>true</allowSelection> <schedulingAllowed>true</schedulingAllowed> </workflowDescriptor> |
...
When a node in the IFD is selected with a workflow which has the allowSelection option set true, the GUI will look like this:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
<explorerTask name="Forecasts"> <taskClass>nl.wldelft.fews.gui.plugin.selection.SegmentSelectionDialog</taskClass> <toolbarTask>false<<predefinedDisplay>topology tree</predefinedDisplay> <toolbarTask>false</toolbarTask> <menubarTask>false</menubarTask> <toolWindow>true</toolWindow> <loadAtStartup>true</loadAtStartup> </explorerTask> |
Trim Output
A boolean option <trimOutput> is available within transformations. When true, missing values at the start and end of the output will be removed before writing the data to the database. This can prevent existing values to be overwritten with missings.
Forecast Loop
For some transformations it is possible to define a forecast loop by configuring a <forecastLoopSearchPeriod>. This means that the transformation will be run for each forecast found within that period.
...
For the most recent development version see the xsd schema at httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/transformationTypes.xsd
Available since stable build 2014.01: