What |
nameofinstance.xml |
|---|---|
Description |
Configuration for the new version of the transformation module |
schema location |
http://fews.wldelft.nl/schemas/version1.0/transformationModule.xsd |
Entry in ModuleDescriptors |
<moduleDescriptor id="TransformationModule"> |
Transformation Module Configuration (New Version)
The Transformation module is a general-purpose module that allows for generic transformation and manipulation of time series data. The module may be configured to provide for simple arithmetic manipulation, time interval transformation, shifting the series in time etc, as well as for applying specific hydro-meteorological transformations such as stage discharge relationships etc.
A new version of the FEWS Transformation Module is currently under construction. The new version is much more easy to configure than the old version. The new version uses a new schema for configuration, which is explained below (TODO). Currently only a few of the old transformations are available in the new transformation module. All of the old transformations will be made available in the new module, as well as several new transformations.
List of transformations
This list will be continuously updated to show the current status of the transformations available in the new transformation module.
Group |
Transformation |
Code completed? |
Test completed? |
Comment |
|
|---|---|---|---|---|---|
adjust |
adjustStage |
|
|
New feature |
|
adjust |
adjustQ |
|
|
New feature |
|
adjust |
|
|
New feature |
||
adjust |
|
|
New feature |
||
adjust |
adjustTide |
|
|
New feature |
|
aggregation |
instantaneous |
|
|
|
|
aggregation |
accumulative |
|
|
|
|
aggregation |
instantaneousToMean |
|
|
|
|
aggregation |
meanToMean |
|
|
|
|
aggregation |
seasonal |
|
|
|
|
altitude |
zero degree altitude |
|
|
|
|
altitude |
elevation lapse |
|
|
|
|
conditional |
multiple |
|
|
This allows definition of multiple rule groups, combined with logical operators (from the current lookup module) |
|
conversion |
datum |
|
|
(New feature) This includes an enumeration option called direction local/global |
|
disaggregation |
accumulative |
|
|
|
|
disaggregation |
instantaneous |
|
|
|
|
disaggregation |
meanToInstantaneous |
|
|
|
|
disaggregation |
meanToMean |
|
|
|
|
disaggregation |
weights |
|
|
|
|
discharge stage |
power |
|
|
Uses the same coefficient sets as stage discharge power. |
|
discharge stage |
table |
|
|
Uses the same coefficient sets (for table entries) as stage discharge table. |
|
evaporation |
penmanNorthEast |
|
|
should be renamed to the specific type of penman equation |
|
filter |
moving window |
|
|
contains options average/maximum/minimum |
|
filter |
block |
|
|
contains options average/maximum/minimum |
|
interpolation serial |
block |
|
|
|
|
interpolation serial |
default |
|
|
|
|
interpolation serial |
extrapolate base |
|
|
Extrapolates by adding future values that start at the last data value and linearly approach a given base value. |
|
interpolation serial |
extrapolate constant |
|
|
Extrapolates by filling the gap between the start of the input time series and the first data value or by filling the gap between the last data value and the end of the input time series or by filling both of these gaps. |
|
interpolation serial |
extrapolate exponential |
|
|
New feature: Extrapolates by adding future values that start at the last data value and exponentially approach a given base value. |
|
interpolation serial |
extrapolate persistence |
|
|
Extrapolates by adding future values that start with the last moving average value and exponentially approach zero. This can be used together with the filterMoving average and userSimple transformations to reproduce the old originalAndPersistence and movingAverageAndPersistence transformations. |
|
interpolation serial |
extrapolate trend |
|
|
Should contain option on how to extrapolate: persistence trend/running mean trend |
|
interpolation serial |
linear |
|
|
|
|
interpolation spatial |
average |
|
|
|
|
interpolation spatial |
bilinear |
|
|
|
|
interpolation spatial |
closest distance |
|
|
|
|
interpolation spatial |
kriging |
|
|
|
|
interpolation spatial |
sum |
|
|
|
|
interpolation spatial |
inputAverageTimesOutputArea |
|
|
|
|
interpolation spatial |
inverse distance |
|
|
(Uses Fortran 90 code) |
|
interpolation spatial |
sum |
|
|
|
|
interpolation spatial |
triangulation |
|
|
(Uses Fortran 90 code) Used to be called Renka & Cline |
|
interpolation spatial |
weighted |
|
|
This is the old weighted average approach used for rainfall. |
|
lookup |
simple |
|
|
should be called oneDimensional. From the current lookup module |
|
lookup |
twoDimensionalLookup |
|
|
should be called twoDimensional. New feature |
|
lookup |
twoDimensionalTable |
|
|
From the current lookup module |
|
merge |
simple |
|
|
merge data hierarchy |
|
merge |
toggle |
|
|
|
|
merge |
mean |
|
|
New feature |
|
merge |
synoptic |
|
|
This needs a true false element to indicate if the smaller or the larger quantities prevail (new feature) |
|
merge |
forecasts |
|
|
equivalent to sampleForecasts - this is a new feature. |
|
moisture |
soil moisture |
|
|
unit test not yet available |
|
moisture |
optical depth |
|
|
unit test not yet available |
|
precipitation |
rainfallEvent |
|
|
writes median value of event (Enumeration of 2 to indicate where to write the data) |
|
precipitation |
rainfallEventLength |
|
|
writes length of event (Enumeration of 2 to indicate where to write the data) |
|
precipitation |
tipping bucket |
|
|
|
|
profile |
timeSeries |
|
|
Converts a profile to a time series. |
|
profile |
harmonics |
|
|
Converts an input profile consisting of harmonic components (a Fourier series) to a time series. |
|
review |
stageReview |
|
|
New feature |
|
review |
tidal balanceReview |
|
|
New feature |
|
sample |
historical |
|
|
|
|
sample |
forecasts |
|
|
This samples the first n values from a forecast - somewhat similar to the merge-forecast function |
|
sample |
equidistant |
|
|
converts an equidistant series to another equidistant series or to a non-equidistant series |
|
sample |
non-equidistant |
|
|
converts a non-equidistant series to another non-equidistant series or to an equidistant series |
|
selection |
peaks |
|
|
|
|
selection |
independent peaks |
|
|
previously called selectWithinHorizontalgaps |
|
selection |
lows |
|
|
|
|
selection |
independent lows |
|
|
|
|
selection |
maximum |
|
|
|
|
selection |
minimum |
|
|
|
|
stage discharge |
power |
|
|
Uses the same coefficient sets as discharge stage power. |
|
stage discharge |
power backwater |
|
|
Backwater correction (new feature). |
|
stage discharge |
power unsteady |
|
|
Unsteady flow (new feature). |
|
stage discharge |
parabolic |
|
|
New feature |
|
stage discharge |
table |
|
|
Uses the same coefficient sets (for table entries) as discharge stage table. |
|
statisticsSummary |
sum, max, etc. |
|
|
Input is one time series. Output is one value. All values in the input series within the transformation run period are used to get a statistic output value. This covers the case in the old transformations for which isBlockFunction is false and no season is specified. |
|
statisticsSerial |
sum, max, etc. |
|
|
These are aggregation functions. Input is one time series. Output is one time series. For each output time all input values within a certain 'aggregation period' are used to get a statistic output value. This is used to get e.g. the mean temperature per day for every day in the input time series. This covers the case in the old transformations for which isBlockFunction is true. Together with the (still to implement) repeating timeStep types this also covers the case in the old transformations for which isBlockFunction is false, a season is specified and seasonal is true. The aggregation periods of different output times can overlap, therefore it is also possible to calculate e.g. moving average. |
|
statisticsPeriodical |
sum, max, etc. |
|
|
These are the seasonal stats - used to define e.g. mean monthly temp. This is e.g. used in climatology to get e.g. the mean temperature in January over the last 100 years. This covers the case in the old transformations for which isBlockFunction is false, a season is specified and seasonal is false. |
|
statisticsEnsemble |
sum, max, etc. |
|
|
Input is an ensemble with multiple time series. Output is one time series. Per time step the ensemble values for that time step are used to get a statistic output value. |
|
statisticsSpatial |
sum, max, etc. |
|
|
New feature. Input is a grid time series. Output is a scalar time series. Per time step all grid cell values for that time step are used to get a statistic output value. |
|
statisticsChildrenLocations |
sum, max, etc. |
|
|
New feature. Input is multiple time series. Output is multiple time series. For each output series (with a parent location) the corresponding child locations are searched in the input series and used to get a statistic output value, per time step. |
|
structure |
general weir fixed height |
|
|
|
|
structure |
general weir variable height |
|
|
|
|
structure |
flatV weir |
|
|
Enumeration to indicate simple or crest tapping |
|
structure |
flatV weir backwater |
|
|
New feature |
|
structure |
crump weir |
|
|
Enumeration to indicate simple or crest tapping |
|
structure |
crump weir backwater |
|
|
New feature |
|
structure |
|
|
|
||
structure |
|
|
|
||
structure |
|
|
|
||
structure |
|
|
|
||
time shift |
constant |
|
|
|
|
time shift |
variable |
|
|
|
|
time shift |
length |
|
|
|
|
user |
|
|
free format expression |
||
user |
|
|
This can be used for periodic output - the range can be defined, e.g 0-1, 0-360 or 0-100 |
Configuration details (TODO)
When available as configuration on the file system, the name of an XML file for configuring an instance of the transformation module called for example TransformHBV_Inputs may be:
TransformHBV_Inputs 1.00 default.xml
TransformHBV_Inputs |
File name for the TransformHBV_Inputs configuration. |
1.00 |
Version number |
default |
Flag to indicate the version is the default configuration (otherwise omitted). |
The configuration for the transformation module consists of two parts: transformation configuration files in the Config/ModuleConfigFiles directory and coefficient set configuration files in the Config/CoefficientSetsFiles directory.
In a transformation configuration file one or more transformations can be configured. See List of transformations for the available types of transformation. Some transformations require coefficient sets in which given coefficients are defined. For a given transformation that requires a coefficient set there are different ways of defining the coefficient set in the configuration. One way is to specify an embedded coefficient set in the transformation configuration itself. Another way is to put a reference in the transformation configuration. This reference consists of the name of a separate coefficient set configuration file and the id of a coefficient set in that file.
Both the transformations and coefficient sets can be configured to be time dependent. This can be used for instance to define a given coefficient value to be 3 from 1 January 2008 to 1 January 2009, and to be 4 from 1 January 2009 onwards. This can be done by defining multiple periodCoefficientSets, each one with a different period, as in the following xml example.
<periodCoefficientSet>
<period>
<startDateTime date="2008-01-01" time="00:00:00"/>
<endDateTime date="2009-01-01" time="00:00:00"/>
</period>
<structure>
<pumpFixedDischarge>
<discharge>3</discharge>
</pumpFixedDischarge>
</structure>
</periodCoefficientSet>
<periodCoefficientSet>
<period>
<validAfterDateTime date="2009-01-01"/>
</period>
<structure>
<pumpFixedDischarge>
<discharge>4</discharge>
</pumpFixedDischarge>
</structure>
</periodCoefficientSet>
If a date is specified without a time, then the time is assumed to be 00:00:00, so <validAfterDateTime date="2009-01-01"/> is the same as <validAfterDateTime date="2009-01-01" time="00:00:00"/>. To specify dates and times in a particular time zone use the optional time zone element at the beginning of a transformations or a coefficient sets configuration file, e.g. <timeZone>GMT+5:00</timeZone>. Then all dates and times in that configuration file are in the defined time zone. If no time zone is defined, then dates and times are in GMT. Note: 2008-06-20 11:33:00 in time zone GMT+5:00 is physically the same time as 2008-06-20 06:33:00 in GMT.
If for a given transformation there are different coefficientSets configured for different periods in time, then the following rule is used. The start of a period is always inclusive. The end of a period is exclusive if another period follows without a gap in between, otherwise the end of the period is inclusive. If for example there are three periodCoefficientSets defined (A, B and C), each with a different period, as in the following xml example. Then at 2002-01-01 00:00:00 periodCoefficientSet A is valid. At 2003-01-01 00:00:00 periodCoefficientSet B is valid since the start of the period is inclusive. At 2004-01-01 00:00:00 periodCoefficientSet B is still valid, since there is a gap after 2004-01-01 00:00:00. At 2011-01-01 00:00:00 periodCoefficientSet C is valid, since no other periods follow (the period of C is the last period in time that is defined). This same rule applies to time-dependent transformations.
<periodCoefficientSet>
<!-- periodCoefficientSet A -->
<period>
<startDateTime date="2002-01-01" time="00:00:00"/>
<endDateTime date="2003-01-01" time="00:00:00"/>
</period>
...
</periodCoefficientSet>
<periodCoefficientSet>
<!-- periodCoefficientSet B -->
<period>
<startDateTime date="2003-01-01" time="00:00:00"/>
<endDateTime date="2004-01-01" time="00:00:00"/>
</period>
...
</periodCoefficientSet>
<periodCoefficientSet>
<!-- periodCoefficientSet C -->
<period>
<startDateTime date="2010-01-01" time="00:00:00"/>
<endDateTime date="2011-01-01" time="00:00:00"/>
</period>
...
</periodCoefficientSet>
