Page History

Accumulative

Input

• inputVariable. (Be careful with using non equidistant input data, this transformation is not designed for that and could give unexpected output values. In case of non-equidistant data the Statistics Serial Sum Transformation might be more suitable)

Options

• validationRules. It is possible to define a set of validationRules. With these validation rules it is possible to define a set of criteria which determines the outputflag of the calculated value based on the number of missing values and/or doubtfulls doubtfuls values counted in the input values.

Output

• outputVariable.
  Please note that that the timestep of the output variable needs to have the same "hour" value. E.g. aggregation from daily to monthly timestep works when both are defined for 00:00 GMT, but not when one is set of 08:00 GMT and the other for 00:00 GMT. This typically is only an issue with user-defined timesteps. The default timesteps of Delft-FEWS (or "units") for day, week and year all have a timestamp at midnight.

Description

This transformation performs an aggregation from an instantaneous time series to an aggregated time series. This procedure sums the values of the input timeseries that are within the aggregation period. If no aggregation period is configured, then the aggregation period is equal to the period between the current output time and the previous output time. Alternatively the aggregation period can be configured in the time series set of the output variable. In that case the aggregation period is relative to the current output time and aggregation periods for different output times are allowed to overlap. Using overlapping aggregation periods it is possible to use this transformation to calculate a moving sum. If one of the input values is missing or unreliable the output is missing.

The table below shows an example of accumulating 6-hourly values to daily values using this transformation.

The figure below shows original 15 minute data and the aggregated hourly data using the accumulative function:

Validation rules

The concept of the validation rules was introduced as a solution for a common problem in operational situations when using aggregation transformations. When for example a yearly average was computed a single missing value in the input values would cause that the yearly average was also a missing value.

The validation rules provide a solution for these types of situations.

The validation rules are optional in the configuration and can be used to define the outputflag and the custom flagsource of the output value based on the number of missing values/unreliables values and/or the number of doubtfull values in the used input values.

With these rules it is possible to define for example that the output of the transformation is reliable if less than 10% of the input is unreliable and/or missing and that if this percentage above 10% that in that case the output should be a missing value.

It is important to note that input values which are missing and input values which are marked as unreliable are treated the same. Both are seen as missing values by the validation rules.

This prevents that a single missing value in the input will lead to a missing value in the aggregated output value.

Below the configuration of the basic example which was described above.

				<validationRule>
					<inputMissingPercentage>10</inputMissingPercentage>
					<outputValueFlag>reliable</outputValueFlag>
				</validationRule>
				<validationRule>
					<inputMissingPercentage>100</inputMissingPercentage>
					<outputValueFlag>missing</outputValueFlag>
				</validationRule>

The configured validation rules are applied in the following way. The first validation rules is applied first. In the example above the first rule is that if 10% or less of the input is missing (or unreliable) that the output flag will be set the reliable. If the input doesn't meet the criteria for the first rule the transformation module will try to apply the second rule. In this case the second rule will always apply because a percentage 100% is configured.

This is a recommended way of configuring the validation rules. By default if validation rules are configured and none of the configured rules are valid the output will be set to missing. But for the users of the system it is more understandable if the behaviour of the aggregation is configured instead of a hard-coded fallback mechanism in the software.

To explain the validation rules a bit more a more difficult example will explained. Let's say that we would like configure our aggregation in such a way that the following rules are applied:

1 if the percentage of missing and/or unreliable values is less than 15% the output should be reliable.
2 if the percentage of missing values is less than 40% the output should be doubtfull.
3 in all other cases the output should be a missing value.

Below shows a configuration example if the rules above were implemented.

See this wiki Validationrules for a general explanation, and below for a config example including validationrules.

Configuration example

Below a configuration example which has proven itself in an operational context.

The idea behind it is that if the majority of the input is doubtful, the output should be doubtful and if the majority of the input is missing/unreliable the output should be missing.

Splitting into quartiles turned out to be most intuitive, distinctive and least complex.

The outputCustomFlagSourceId follows the naming convention of the amount of doubtful and missing/unreliable quartiles. For example “d3u1” stands for 75% doubtful and 25% missing/unreliable. This way the reason for the quality flag is easy to understand when shown in FEWS.

<?xml version="1.0" encoding="UTF-8"?>
<transformation id="DebietBerekeningDag">
	<aggregation>
		<accumulative>
			<inputVariable>
				<timeSeriesSet>
					<moduleInstanceId>DebietBerekening</moduleInstanceId>
					<valueType>scalar</valueType>
					<parameterId>Q.berekend.15min</parameterId>
					<locationSetId>HHNK_TMX_debiet</locationSetId>
					<timeSeriesType>external historical</timeSeriesType>
					<timeStep unit="minute" multiplier="15"/>
					<relativeViewPeriod unit="day" start="-300" end="0"/>
					<readWriteMode>read only</readWriteMode>
					<synchLevel>1</synchLevel>
				</timeSeriesSet>
			</inputVariable>
			<validationRule>
				<inputDoubtfulPercentage>0</inputDoubtfulPercentage>
				<inputMissingPercentage>0</inputMissingPercentage>
				<outputValueFlag>reliable</outputValueFlag>
				<outputCustomFlagSourceId>d0u0</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>25</inputDoubtfulPercentage>
				<inputMissingPercentage>0</inputMissingPercentage>
				<outputValueFlag>reliable</outputValueFlag>
				<outputCustomFlagSourceId>d1u0</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>50</inputDoubtfulPercentage>
				<inputMissingPercentage>0</inputMissingPercentage>
				<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d2u0</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>75</inputDoubtfulPercentage>
				<inputMissingPercentage>0</inputMissingPercentage>
				<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d3u0</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>100</inputDoubtfulPercentage>
				<inputMissingPercentage>0</inputMissingPercentage>
				<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d4u0</outputCustomFlagSourceId>
			</validationRule>

				<validationRule>
					<inputMissingPercentage>15<<inputDoubtfulPercentage>25</inputDoubtfulPercentage>
				<inputMissingPercentage>25</inputMissingPercentage>
					<outputValueFlag>reliable</outputValueFlag>
				<outputCustomFlagSourceId>d0u1</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>50</inputDoubtfulPercentage>
				<inputMissingPercentage>25</inputMissingPercentage>
				<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d1u1</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
					<inputMissingPercentage>40<<inputDoubtfulPercentage>75</inputDoubtfulPercentage>
				<inputMissingPercentage>25</inputMissingPercentage>
				<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d2u1</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>100</inputDoubtfulPercentage>
				<inputMissingPercentage>25</inputMissingPercentage>
				<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d3u1</outputCustomFlagSourceId>
			</validationRule>
				<validationRule>
				<inputDoubtfulPercentage>50</inputDoubtfulPercentage>
		<inputMissingPercentage>100<		<inputMissingPercentage>50</inputMissingPercentage>
					<outputValueFlag>missing<<outputValueFlag>doubtful</outputValueFlag>
				<outputCustomFlagSourceId>d0u2</outputCustomFlagSourceId>
			</validationRule>

In some cases in it one would like to differ between situations in which for example the output was marked as reliable. In the example above if all of the input values were reliable the output is marked reliable. But if for example 10% of the input values were unreliable the output is also marked as reliable.

It would be nice if the user of the system would be able to see in the GUI of FEWS why the input was marked reliable.

To make this possible the concept of the custom flag source is added the validation rules. In addition to configuring an output flag it is also possible to configure a custom flag source. In the table of the Timeseriesdialog the custom flag source can be visible by

Below an example in which w

Configuration example

	<transformation id="aggregation accumulative">
		<aggregation>
			<accumulative>
				<inputVariable>
					<timeSeriesSet>
						<moduleInstanceId>ImportTelemetry</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>H.obs</parameterId>
						<locationSetId>hydgauges</locationSetId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="minute" multiplier="15"/>
						<relativeViewPeriod unit="day" startOverrulable="true" start="-7" end="0"/>
						<readWriteMode>read only</readWriteMode>
						<delay unit="minute" multiplier="0"/>
					</timeSeriesSet>
				</inputVariable>
				<outputVariable>
				<validationRule>
				<inputDoubtfulPercentage>75</inputDoubtfulPercentage>
				<inputMissingPercentage>50</inputMissingPercentage>
				<outputValueFlag>missing</outputValueFlag>
				<outputCustomFlagSourceId>d1u2</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>100</inputDoubtfulPercentage>
				<inputMissingPercentage>50</inputMissingPercentage>
				<outputValueFlag>missing</outputValueFlag>
				<outputCustomFlagSourceId>d2u2</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>75</inputDoubtfulPercentage>
				<inputMissingPercentage>75</inputMissingPercentage>
				<outputValueFlag>missing</outputValueFlag>
				<outputCustomFlagSourceId>d0u3</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>100</inputDoubtfulPercentage>
				<inputMissingPercentage>75</inputMissingPercentage>
				<outputValueFlag>missing</outputValueFlag>
				<outputCustomFlagSourceId>d1u3</outputCustomFlagSourceId>
			</validationRule>
			<validationRule>
				<inputDoubtfulPercentage>100</inputDoubtfulPercentage>
				<inputMissingPercentage>100</inputMissingPercentage>
				<outputValueFlag>missing</outputValueFlag>
				<outputCustomFlagSourceId>d0u4</outputCustomFlagSourceId>
			</validationRule>
			<outputVariable>
				<timeSeriesSet>
						<moduleInstanceId>Aggregate_Historic<<moduleInstanceId>DebietBerekening</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>accumulative<<parameterId>Q.berekend.15min</parameterId>
						<locationSetId>hydgauges<<locationSetId>HHNK_TMX_debiet</locationSetId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="hourminute" multiplier="115"/>
						<relativeViewPeriod unit="day" startOverrulable="true" start="-7300" end="0"/>
						<readWriteMode>add<readWriteMode>read originals<only</readWriteMode>
						<synchLevel>1</synchLevel>
					</timeSeriesSet>
				</outputVariable>
			</accumulative>
		</aggregation>
	</transformation>