20 Transformation Module (Improved schema)

Contents

Transformation Module Configuration (New Version)

The Transformation module is a general-purpose module that allows for generic transformation and manipulation of time series data. The module may be configured to provide for simple arithmetic manipulation, time interval transformation, shifting the series in time etc, as well as for applying specific hydro-meteorological transformations such as stage discharge relationships etc.

An improvement version of the FEWS Transformation Module is currently under construction. The new version is much more easy to configure than the old version. The new version uses a new schema for configuration, also several new transformations are added.

Configuration    

When available as configuration on the file system, the name of an XML file for configuring an instance of the transformation module called for example TransformHBV_Inputs may be:

TransformHBV_Inputs 1.00 default.xml.

The configuration for the transformation module consists of two parts: transformation configuration files in the Config/ModuleConfigFiles directory and coefficient set configuration files in the Config/CoefficientSetsFiles directory.

In a transformation configuration file one or more transformations can be configured. Some transformations require coefficient sets in which given coefficients are defined. For a given transformation that requires a coefficient set there are different ways of defining the coefficient set in the configuration. One way is to specify an embedded coefficient set in the transformation configuration itself. Another way is to put a reference in the transformation configuration. This reference consists of the name of a separate coefficient set configuration file and the id of a coefficient set in that file.

Both the transformations and coefficient sets can be configured to be time dependent. This can be used for instance to define a given coefficient value to be 3 from 1 January 2008 to 1 January 2009, and to be 4 from 1 January 2009 onwards. This can be done by defining multiple periodCoefficientSets, each one with a different period, as in the following xml example.

<periodCoefficientSet>
  <period>
    <startDateTime date="2008-01-01" time="00:00:00"/>
    <endDateTime date="2009-01-01" time="00:00:00"/>
  </period>
  <structure>
    <pumpFixedDischarge>
      <discharge>3</discharge>
    </pumpFixedDischarge>
  </structure>
</periodCoefficientSet>
<periodCoefficientSet>
  <period>
    <validAfterDateTime date="2009-01-01"/>
  </period>
  <structure>
    <pumpFixedDischarge>
      <discharge>4</discharge>
    </pumpFixedDischarge>
  </structure>
</periodCoefficientSet>

If a date is specified without a time, then the time is assumed to be 00:00:00, so <validAfterDateTime date="2009-01-01"/> is the same as <validAfterDateTime date="2009-01-01" time="00:00:00"/>. To specify dates and times in a particular time zone use the optional time zone element at the beginning of a transformations or a coefficient sets configuration file, e.g. <timeZone>GMT+5:00</timeZone>. Then all dates and times in that configuration file are in the defined time zone. If no time zone is defined, then dates and times are in GMT. Note: 2008-06-20 11:33:00 in time zone GMT+5:00 is physically the same time as 2008-06-20 06:33:00 in GMT.

If for a given transformation there are different coefficientSets configured for different periods in time, then the following rule is used. The start of a period is always inclusive. The end of a period is exclusive if another period follows without a gap in between, otherwise the end of the period is inclusive. If for example there are three periodCoefficientSets defined (A, B and C), each with a different period, as in the following xml example. Then at 2002-01-01 00:00:00 periodCoefficientSet A is valid. At 2003-01-01 00:00:00 periodCoefficientSet B is valid since the start of the period is inclusive. At 2004-01-01 00:00:00 periodCoefficientSet B is still valid, since there is a gap after 2004-01-01 00:00:00. At 2011-01-01 00:00:00 periodCoefficientSet C is valid, since no other periods follow (the period of C is the last period in time that is defined). This same rule applies to time-dependent transformations.

<periodCoefficientSet>
  <!-- periodCoefficientSet A -->
  <period>
    <startDateTime date="2002-01-01" time="00:00:00"/>
    <endDateTime date="2003-01-01" time="00:00:00"/>
  </period>
  ...
</periodCoefficientSet>
<periodCoefficientSet>
  <!-- periodCoefficientSet B -->
  <period>
    <startDateTime date="2003-01-01" time="00:00:00"/>
    <endDateTime date="2004-01-01" time="00:00:00"/>
  </period>
  ...
</periodCoefficientSet>
<periodCoefficientSet>
  <!-- periodCoefficientSet C -->
  <period>
    <startDateTime date="2010-01-01" time="00:00:00"/>
    <endDateTime date="2011-01-01" time="00:00:00"/>
  </period>
  ...
</periodCoefficientSet>

Run transformations for a set of selected locations

In some cases it is usefull to run a transformation only for a specific set of locations. For example when the entire workflow has already ran and there is only a change at a specific location. This situation can occur, for example, when a water level has been editted or when the configuration is changed.

In this case the workflow can skip the calculations for the unchanged locations. The main benefit of this approach is that it saves a lot of processing time.

This functionality is now available in FEWS.However it is important to understand that this functionality cannot be used in all workflows. The functionality can be applied for transformations only. It cannot be used for running models or secondary validations. When a transformation is started for a location selection than the transformation will only start when the location of one of the input time series is selected or the location of the output time series is selected. When a transformation has created output for a location which was not selected by the user than this location whill be added to the selection.

It is possible to run a workflow for a selected set of locations from the IFD, the task dialog and the manual forecast dialog.

It is also important to note that this functionality is limited to transformations where the locations of the input time series and the output location are the same. This is typicaly the case for interpolations, aggregations etc.

This functionality cannot be used for spatial transformations. This will be explained with a simple example.

In this example we have a workflow with 2 transformations. The first transformation consists of a interpolation. In the second transformation a closest distance transformation is done. The user wants to calculate the output for a certain output location of the closest distance transformation. When the workflow is started, the interpolation transformation will be started first. However at this point FEWS should know which locations will used as input when the time series for the selected locations will be calculated, because these are the locations for which a interpolation needs to be done. The other calculations can be skipped. Although it is possible to derive such information by analyzing the flow of the transformation it is decided at this point not to implement this. This means that this functionality cannot be applied for this type of tranformations.

This is reason why it is only possible to run a set of transformations for a location selection for tranformations in which the input locations are the same as the output locations. There is one expection for this rule. For the statisticRelatedLocation-transformations the functionality is also supported. When such a transformation is started and the outputlocation has a relation with 1 or more inputlocations and the either the outputlocation is selected or one the inputlocations than the selected locations will be extended with the inputlocations and the outputlocations.

In addition to the above, this functionality can only be used for non-forecast workflows. Typically this functionality should be used for pre-processing of post-processing.

Therefore it is by default not possible to run a workflow for a specific location selection. This is only possible when in the workflowdescriptors the option allowSelection is to true. This option should only be set to true when the configurator has checked that the workflow is suitable for running for a specific location.

Implementation plan

The following steps should be followed when this functionality is implemented.
1) Decide in which situations this functionality is needed
2) Make a list of the workflows which need to run in this type of situations
3) Ensure that the workflow only consists of transformations for which this functionality can be used
4) Move transformations or other parts of the workflow which are suitable for this type of operations to another workflow
5) Set the option allowSelection to true in the workflow descriptor for the workflow which can be used for selection specific calculations