Versions Compared

Old Version 35

changes.mady.by.user Andre Grijze

Saved on

compared with

New Version 36

changes.mady.by.user Klaas-Jan van Heeringen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

 

Original series

Result

Date/Time

Value

Value

01-01-2007 00:00

1,.00

 

01-01-2007 06:00

2,.00

 

01-01-2007 12:00

3,.00

 

01-01-2007 18:00

4,.00

 

02-01-2007 00:00

5,.00

14,.00

02-01-2007 06:00

6,.00

 

02-01-2007 12:00

NaN

 

02-01-2007 18:00

8,.00

 

03-01-2007 00:00

9,.00

NaN

03-01-2007 06:00

10,.00

 

The figure below shows original 15 minute data and the aggregated hourly data using the accumulative function:

...

Below shows a configuration example in which the rules above are implemented.

Code Block
				<validationRule>
					  <inputMissingPercentage>15</inputMissingPercentage>
					  <outputValueFlag>reliable</outputValueFlag>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>40</inputMissingPercentage>
					  <outputValueFlag>doubtful</outputValueFlag>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>100</inputMissingPercentage>
					  <outputValueFlag>missing</outputValueFlag>
				</validationRule>

The example shows that in total 3 validation rules were needed. The first rule checks if less than 15% of the input is missing/unreliable. If this is not the case than it will be checked if the second rule can be applied. The second rule states that if less than 40% of the input is missing that in that case the output flag should be set to doubtful. The last rule takes care of all the other situations. Note that it has a percentage configured of 100%. Which means that this rule will be applied. However because 2 rules are defined above this rule FEWS will always try to apply these rules first before applying this rule.

...

Below an example in which the output is reliable when there are no missing values in the input and when the percentage if missing values is less than 15%. However in the first case the output doesn't get a custom flagsource assigned while in the second case the output gets a custom flagsource assigned which is visible in the GUI to indicate that a output value was calculated but that missing values were found in the input.

Code Block
	<validationRule>
					  <inputMissingPercentage>0</inputMissingPercentage>
					  <outputValueFlag>reliable</outputValueFlag>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>15</inputMissingPercentage>
					  <outputValueFlag>reliable</outputValueFlag>
					  <outputCustomFlagSourceId>CA</outputCustomFlagSourceId>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>40</inputMissingPercentage>
					  <outputValueFlag>doubtful</outputValueFlag>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>100</inputMissingPercentage>
					  <outputValueFlag>missing</outputValueFlag>
	</validationRule>

Finally it is also possible to define validation rules based on the number of doubtful values in the input. It is important to note that missing values and unrealiable values found in the input are also counted as doubtfull values. It is even possible to define validation rules based on a combination of an allowed percentage of unreliable/missing values and doubtful values. The sequence of applying the rules is also in this case the order in which the rules are configured. The first rule which applies to the
current situation is used.

...

Below a configuration example

Code Block
				<validationRule>
					  <inputDoubtfulPercentage>10</inputDoubtfulPercentage>
					  <inputMissingPercentage>0</inputMissingPercentage>
					  <outputValueFlag>reliable</outputValueFlag>
				</validationRule>
				<validationRule>
					  <inputDoubtfulPercentage>30</inputDoubtfulPercentage>
					  <inputMissingPercentage>0</inputMissingPercentage>
					  <outputValueFlag>doubtful</outputValueFlag>
					  <outputCustomFlagSourceId>D1</outputCustomFlagSourceId>
				</validationRule>
				<validationRule>
					  <inputDoubtfulPercentage>60</inputDoubtfulPercentage>
					  <inputMissingPercentage>0</inputMissingPercentage>
					  <outputValueFlag>doubtful</outputValueFlag>
					  <outputCustomFlagSourceId>D2</outputCustomFlagSourceId>
				</validationRule>
				<validationRule>
					  <inputDoubtfulPercentage>100</inputDoubtfulPercentage>
					  <inputMissingPercentage>0</inputMissingPercentage>
					<outputValueFlag>doubtful<  <outputValueFlag>doubtful</outputValueFlag>
					  <outputCustomFlagSourceId>D3</outputCustomFlagSourceId>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>15</inputMissingPercentage>
					  <outputValueFlag>reliable</outputValueFlag>
					  <outputCustomFlagSourceId>CA</outputCustomFlagSourceId>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>40</inputMissingPercentage>
					<outputValueFlag>doubtful<  <outputValueFlag>doubtful</outputValueFlag>
				</validationRule>
				<validationRule>
					  <inputMissingPercentage>100</inputMissingPercentage>
					  <outputValueFlag>missing</outputValueFlag>
				</validationRule>

The explanation above gave an good idea of the possibilities of the use of the validation rules.

...

Code Block
<validationRule>
	  <inputMissingPercentage>40</inputMissingPercentage>
	  <outputValueFlag>doubtful</outputValueFlag>
</validationRule>

...

Code Block
<validationRule>
	  <inputMissingPercentage>100</inputMissingPercentage>
	  <outputValueFlag>missing</outputValueFlag>
</validationRule>

...

Code Block
<validationRule>
    <inputDoubtfulPercentage>30</inputDoubtfulPercentage>
    <inputMissingPercentage>0</inputMissingPercentage>
    <outputValueFlag>doubtful</outputValueFlag>
    <outputCustomFlagSourceId>D1</outputCustomFlagSourceId>
</validationRule>

...

Configuration example
Code Block
	<transformation id="DebietBerekeningDag">
		  <aggregation>
			<accumulative>
				<inputVariable>
					<timeSeriesSet>
						<moduleInstanceId>DebietBerekening</moduleInstanceId>
						<valueType>scalar</valueType>
						    <accumulative>
      <inputVariable>
        <timeSeriesSet>
          <moduleInstanceId>DebietBerekening</moduleInstanceId>
          <valueType>scalar</valueType>
          <parameterId>Q.berekend.15min</parameterId>
						          <locationSetId>HHNK_TMX_debiet</locationSetId>
						          <timeSeriesType>external historical</timeSeriesType>
						          <timeStep unit="minute" multiplier="15"/>
						          <relativeViewPeriod unit="day" start="-300" end="0"/>
						          <readWriteMode>read only</readWriteMode>
						          <synchLevel>1</synchLevel>
					        </timeSeriesSet>
				      </inputVariable>
				<validationRule>
					      <validationRule>
        <inputDoubtfulPercentage>10</inputDoubtfulPercentage>
					        <inputMissingPercentage>0</inputMissingPercentage>
					        <outputValueFlag>reliable</outputValueFlag>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputDoubtfulPercentage>30</inputDoubtfulPercentage>
					        <inputMissingPercentage>0</inputMissingPercentage>
					        <outputValueFlag>doubtful</outputValueFlag>
					        <outputCustomFlagSourceId>D1</outputCustomFlagSourceId>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputDoubtfulPercentage>60</inputDoubtfulPercentage>
					        <inputMissingPercentage>0</inputMissingPercentage>
					        <outputValueFlag>doubtful</outputValueFlag>
					        <outputCustomFlagSourceId>D2</outputCustomFlagSourceId>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputDoubtfulPercentage>100</inputDoubtfulPercentage>
					        <inputMissingPercentage>0</inputMissingPercentage>
					<outputValueFlag>doubtful</outputValueFlag>
					        <outputValueFlag>doubtful</outputValueFlag>
        <outputCustomFlagSourceId>D3</outputCustomFlagSourceId>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputMissingPercentage>0</inputMissingPercentage>
					        <outputValueFlag>reliable</outputValueFlag>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputMissingPercentage>15</inputMissingPercentage>
					        <outputValueFlag>reliable</outputValueFlag>
					        <outputCustomFlagSourceId>CA</outputCustomFlagSourceId>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputMissingPercentage>40</inputMissingPercentage>
					        <outputValueFlag>doubtful</outputValueFlag>
				      </validationRule>
				<validationRule>
					      <validationRule>
        <inputMissingPercentage>100</inputMissingPercentage>
					        <outputValueFlag>missing</outputValueFlag>
				      </validationRule>
				<outputVariable>
					<timeSeriesSet>
						<moduleInstanceId>DebietBerekening</moduleInstanceId>
						<valueType>scalar</valueType>
						      <outputVariable>
        <timeSeriesSet>
          <moduleInstanceId>DebietBerekening</moduleInstanceId>
          <valueType>scalar</valueType>
          <parameterId>Q.berekend.15min</parameterId>
						          <locationSetId>HHNK_TMX_debiet</locationSetId>
						          <timeSeriesType>external historical</timeSeriesType>
						          <timeStep unit="minute" multiplier="15"/>
						          <relativeViewPeriod unit="day" start="-300" end="0"/>
						          <readWriteMode>read only</readWriteMode>
						          <synchLevel>1</synchLevel>
					        </timeSeriesSet>
				      </outputVariable>
			    </accumulative>
		  </aggregation>
	</transformation>