Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

How to add a custom transformation to my FEWS system?

...

  1. writing the code for the custom tranformationtransformation.
  2. configuring the custom tranformationtransformation,

How to write a custom transformation?

First step is to write the actual code for the tranformationtransformation. Running and debugging the actual transformation can be done by writing unit tests.

...

The code needed for such a transformation would be quite basic and is given below:

Code Block

package example;

import nl.wldelft.fews.openapi.transformationmodule.Calculation;
import nl.wldelft.fews.openapi.transformationmodule.Input;
import nl.wldelft.fews.openapi.transformationmodule.Output;
import nl.wldelft.util.timeseries.Variable;
import org.apache.log4j.Logger;

public class Example1 implements Calculation {
    private static final Logger log = Logger.getLogger(Example1.class.getName());

    @Input
    float option1;

    @Input
    float option2;

    @Input
    Variable input1 = null;

    @Input
    Variable input2 = null;

    @Output
    Variable output = null;

    @Override
    public void calculate() throws Exception {
        if (Float.isNaN(input1.value) || Float.isNaN(input2.value)) return;
        log.info("Input is " + input1.value + " and " + input2.value);
        output.value = (input1.value * option1 + input2.value * option2) / 2;
        log.info("Output is " + output.value);
    }
}

...

Now the code above will explained step by step to understand the details. First note that tranformation transformation implements the interface Calculation.

The interface Calculation has one method void calculate().

Code Block

package nl.wldelft.fews.openapi.transformationmodule;

import nl.wldelft.util.Initializable;
import nl.wldelft.util.TimeZeroConsumer;

public interface Calculation extends Function {

    /**
     * This method is called by the framework to do the actual calculations.
     * The implementing class should get the input from its InputVariable
     * fields and put the calculation output in its OutputVariable fields.
     * The framework will initialize the InputVariable and OutputVariable
     * fields and set the input values in the InputVariables before calling
     * this method and get the output values out of the OutputVariables
     * afterwards.
     *
     * @throws Exception
     */
    void calculate() throws Exception;
}

The method calculate() will contain the actual code for the transformation. The FEWS Transformation framework will automaticly invoke this method.

...

They are both defined as fields in the java-class.

Code Block

@Input
float option1;

@Input
float option2;

@Input
Variable input1 = null;

@Input
Variable input2 = null;

The fields option1 and option2 however dont have any values assigned! How can they have a value during runtime?
The answer is that the FEWS Transformation framework will inject the values at runtime in these variables. The actual values
are described defined in the configuration of the transformation. In the upcoming section, the configuration will be explained in detail.

...

  1. Variable (as is done in the example above)
  2. TimSeriesArrayunmigrated-wiki-markup
  3. Variable\[\]unmigrated-wiki-markup
  4. TimeSeriesArray\[\]

The major difference between the use of the class Variable and TimeSeriesArray is that Variable is used when the output is only dependend on input values

of at the same timestep. For example when the transformation is calculating the discharge from the stage by using a rating curve than the output is only dependend of

...

is dependend on other values at other time steps in the input time series. In this case one should use the TimeSeriesArray.unmigrated-wiki-markup

The same logic applies to the cases when a Variable\[\] or TimeSeriesArray\[\] is used. The difference between a Variable and a Variable\[\] is that when in the configuration severalunmigrated-wiki-markup

timeseries are defined in the timeseriesset config a Variable\[\] is used. For example by configuring a locationSet in the timeSeriesSet. When the timeseriesset only defines a single

timeseries than a Variable is used.

...

In our example the output (just as the input) is defined as a Variable.

Code Block

@Output
Variable output = null;

...

in the input (Note than when the output is defined as a TimeSeriesArray, the input should also be defined as a TimSeriesArrayTimeSeriesArray). The example code we are using, could for example also

be writting with the input and output defined as a TimeSeriesArray.

The example code of such a transformation is given below.

The example code is given below.

Code Block
Code Block

package example;

import nl.wldelft.fews.openapi.transformationmodule.Calculation;
import nl.wldelft.fews.openapi.transformationmodule.Input;
import nl.wldelft.fews.openapi.transformationmodule.Output;
import nl.wldelft.util.timeseries.TimeSeriesArray;
import org.apache.log4j.Logger;

public class Example2 implements Calculation {
    private static final Logger log = Logger.getLogger(Example1.class.getName());

    @Input
    private float option1;

    @Input
    private float option2;

    @Input
    TimeSeriesArray input1 = null;

    @Input
    TimeSeriesArray input2 = null;

    @Output
    TimeSeriesArray output = null;

    @Override
    public void calculate() throws Exception {
        for (int i = 0; i < output.size(); i++) {
            long time = output.getTime(i);

            int index1 = input1.indexOfTime(time);
            if (index1 == -1) continue;
            float value1 = input1.getValue(index1);

            int index2 = input2.indexOfTime(time);
            if (index2 == -1) continue;
            float value2 = input2.getValue(index2);

            if (Float.isNaN(value1) || Float.isNaN(value2)) return;
            log.info("Input is " + value1 + " and " + value2);
            float outputValue = (value1 * option1 + value2 * option2) / 2;
            log.info("Output is " + outputValue);
            output.setValue(i, outputValue);
        }
    }
}

The main difference between the first and the second example is that in the second example the code needs
to have a for loop to calculate the output value for each time step available in the output time series. In the first
example the FEWS transformation framework takes care of the loop.

How to configure a custom transformation?

...

Below you can find an example of a custom fews transformation.

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<transformationModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/transformationModule.xsd" version="1.0">
	        <variable>
		<variableId>input1</variableId>
	       <timeSeriesSet>
                <moduleInstanceId>POFlathead_MergeMAP_UpdateStates</moduleInstanceId>
        <valueType>scalar</valueType>
        <parameterId>MAP<<valueType>scalar</parameterId>valueType>
        <locationId>WGCM8U</locationId>
        <timeSeriesType>simulated historical<<parameterId>MAP</timeSeriesType>parameterId>
                <locationId>WGCM8U</locationId>
                <timeSeriesType>simulated historical</timeSeriesType>
                <timeStep unit="hour" multiplier="6"/>
                <relativeViewPeriod unit="week" start="-520" end="0"/>
         <readWriteMode>add originals</readWriteMode>
      <readWriteMode>add originals</readWriteMode>
       </timeSeriesSet>
	       </variable>
	<variable>
		<variableId>input2</variableId>
		<timeSeriesSet>
        <moduleInstanceId>POFlathead_MergeMAP_UpdateStates</moduleInstanceId>
        <valueType>scalar</valueType>
        <parameterId>MAP</parameterId>
        <locationId>WGCM8L</locationId>
        <timeSeriesType>simulated historical</timeSeriesType>
        <timeStep unit="hour" multiplier="6"/>
           <relativeViewPeriod unit="week" start="-520" end="0"/>
        <readWriteMode>add originals</readWriteMode>
    </timeSeriesSet>
	</variable>
	<variable>
		<variableId>output</variableId>
		<timeSeriesSet>
			<moduleInstanceId>example1</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>MAP</parameterId>
			<locationId>WGCM8</locationId>
			<timeSeriesType>simulated forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="6"/>
			    <relativeViewPeriod unit="week" start="-520" end="0"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<transformation id="Example1">
<custom>
	<userDefined>
		<input>
			<fieldName>input1</fieldName>
			<inputVariable>
				<variableId>input1</variableId>
			</inputVariable>
		</input>
		<input>
			<fieldName>input2</fieldName>
			<inputVariable>
				<variableId>input2</variableId>
			</inputVariable>
		</input>
		<options>
			<float value="0.3" key="option1"></float>
			<float value="0.7" key="option2"></float>
		</options>
		<output>
			<fieldName>output</fieldName>
			<outputVariable>
				<variableId>output</variableId>
			</outputVariable>
		</output>
		<binDir>example</binDir>
		<className>example.Example1</className>
	</userDefined>
</custom>
	</transformation>
</transformationModule>

In this example we showed a custom transformation which uses the example function in the section above.
In the className the name of the implementing class is defined. The binDir section is used to define
the directory which contains the jar(s) with the implementing class and its helper classes.

The section options is used to define optional fields.

Optional fields can be of type:

  • String,
  • int,
  • float,
  • boolean.

The optional fields have a name. By using the name defined in the xml-config a field with the same name is looked up in the java-class.

Secondly a check is done to verify that the type defined int the xml-config is the same as the type defined in the java class. If the check

succeeds the value is injected into the field.

The output section defined the output time series. A output variable (identified by @Output) is searched for in the java class which has the same

name as the name identified in the tag fieldName. If such a variable is found it is linked to the time series configured in the xml file.

The input section defines the input time series. The same logic as is used for the output time series is used here for the output time series.

Using location attributes in a custom transformation

In order to access location attributes in a custom trasformation a couple of things need to be done.

  • The attibuteId's should be passed via the options as string key / value pairs
  • Have \@Input fields with the same name as the key from the string options
  • The custom transformation should implement the interface LocationAttributeValuesProviderConsumer
  • Store the LocationAttributeValuesProvider from the setLocationAttributeValuesProvider method as a field 
  • Get the attribute values from the LocationAttributeValuesProvider via the getXXXValue (single value) or getXXXValues (multivalued) methods

Example config:

Code Block
languagexml
titleExample config for passing attribute id's to custom transformation
<?xml version="1.0" encoding="UTF-8"?>
<transformationModule version="1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/transformationModule.xsd">
	<variable>
		<variableId>input</variableId>
		<timeSeriesSet>
			<moduleInstanceId>UserDefinedFunctionTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Q.m</parameterId>
			<locationSetId>locationAttributeTest</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1"/>
			<relativeViewPeriod unit="day" start="0" end="1"/>
			<readWriteMode>editing visible to all future task runs</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>output</variableId>
		<timeSeriesSet>
			<moduleInstanceId>UserDefinedFunctionTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Q.dis</parameterId>
			<locationSetId>locationAttributeTest</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1"/>
			<relativeViewPeriod unit="day" start="0" end="1"/>
			<readWriteMode>editing visible to all future task runs</readWriteMode>
		</timeSeriesSet>
	</variable>
	<transformation id="userDefinedFunctionTestWithLocationAttributes">
		<custom>
			<userDefined>
				<input>
					<fieldName>input</fieldName>
					<inputVariable>
						<variableId>input</variableId>
					</inputVariable>
				</input>
				<options>
					<string key="stringAttributeID" value="stringAttributeKey"></string>
					<string key="booleanAttributeID" value="booleanAttributeKey"></string>
					<string key="dateTimeAttributeID" value="dateTimeAttributeKey"></string>
					<string key="doubleAttributeID" value="doubleAttributeKey"></string>
				</options>
				<output>
					<fieldName>output</fieldName>
					<outputVariable>
						<variableId>output</variableId>
					</outputVariable>
				</output>
				<className>nl.wldelft.fews.openapi.transformationmodule.CustomTestFunctionWithLocationAttributes</className>
			</userDefined>
		</custom>
	</transformation>
</transformationModule>

Example Java class:

Code Block
languagejava
titleExample Java class for custom transformation processing location attributes
package nl.wldelft.fews.openapi.transformationmodule;

import nl.wldelft.util.timeseries.Variable;

import java.util.Arrays;

public class CustomTestFunctionWithLocationAttributes implements Calculation, LocationAttributeValuesProviderConsumer {
    @Input
    Variable input = null;

    @Input
    String stringAttributeID = null;
    @Input
    String booleanAttributeID = null;
    @Input
    String dateTimeAttributeID = null;
    @Input
    String doubleAttributeID = null;

    @Output
    Variable output = null;

    private LocationAttributeValuesProvider locationAttributeValuesProvider = null;

    @Override
    public void calculate() throws Exception {
        String locationId = input.header.getLocationId();

        // Single value
        locationAttributeValuesProvider.getBooleanValue(locationId, booleanAttributeID);
        locationAttributeValuesProvider.getStringValue(locationId, stringAttributeID);
        locationAttributeValuesProvider.getNumericValue(locationId, doubleAttributeID);
        locationAttributeValuesProvider.getDateTimeValue(locationId, dateTimeAttributeID);

        // Multivalued
        locationAttributeValuesProvider.getBooleanValues(locationId, booleanAttributeID);
        locationAttributeValuesProvider.getStringValues(locationId, stringAttributeID);
        locationAttributeValuesProvider.getNumericValues(locationId, doubleAttributeID);
        locationAttributeValuesProvider.getDateTimeValues(locationId, dateTimeAttributeID);
    }

    @Override
    public void setLocationAttributeValuesProvider(LocationAttributeValuesProvider locationAttributeValuesProvider) {
        this.locationAttributeValuesProvider = locationAttributeValuesProvider;
    }
}

Specifying whether unreliable data should be used as input

Since 2023.02 custom transformations can implement the UseUnreliableInputValueFunction to determine whether or not unreliable data is allowed as input for the transformation.

When this interface is implemented the method useUnreliables() must be specified. A boolean must be returned which defines whether or not unreliable data should be included in the input.

Unfortunately it is not possible to use a configured option to determine if unreliables are allowed, because the usage of unreliables are handled before the configured options are processed. 

Code Block
languagejava
titleExample Java class for custom transformation processing location attributes
package nl.wldelft.fews.openapi.transformationmodule;

import nl.wldelft.util.timeseries.TimeSeriesArray;

public class CustomIncludeUnreliablesTestFunction implements Calculation, UseUnreliableInputValueFunction {

    @Input
    TimeSeriesArray inputSeries = null;
    @Output
    TimeSeriesArray outputSeries = null;

    @Override
    public boolean useUnreliables() {
        return true;
    }

    @Override
    public void calculate() throws Exception {
        for (int i = 0, size = inputSeries.size(); i < size; i++) {
            long time = inputSeries.getTime(i);
            outputSeries.putValue(time, inputSeries.getValue(i));
            int indexOfTime = outputSeries.indexOfTime(time);
            outputSeries.setFlag(indexOfTime, inputSeries.getFlag(i));
        }
    }
}