View Source

Dynamically determines the forecast length or the start time of the run based on timeseries input

Function:	Sets the forecast length or the start of the run

Module Name:	ForecastLengthEstimator
Where to Use?	In a workflow
Why to Use?	To set the length of a forecast or the start of the forecast based on (external) timeseries.
Description:	The forecastLengthEstimator is a module that can be used at the start of a workflow the set the length of the operations in the other modules in that workflow.
Preconditions:	the endoverrulable attribute in the relative view period in the time series sets must be set to true in all modules you want to apply the forecast length to
Outcome(s):
Scheendump(s):	link to attached screendump(s) for displays only
Remark(s):
Available since:	DelftFEWS200803

Overview

The forecastLengthEstimator is a module that can be used at the start of a workflow to set the length of the operations in the other modules in that workflow. As most models cannot handle gaps in the input data, this option can be useful if you want to run a hydrological model only with the data available and thus avoid e.g. extrapolating the meteorological forecast data.

Forecast time series

A configuration example of the forecast length estimator using an external forecast time series is given below:

<?xml version="1.0" encoding="UTF-8"?>
<forecastLengthEstimator xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd">
	<externalForecastTimeSeries>
			<moduleInstanceId>ImportCOSMO2</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>P.fg</parameterId>
			<locationId>COSMO2</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="1"/>
			<relativeViewPeriod unit="hour" start="0" end="30"/>
			<readWriteMode>read only</readWriteMode>
	</externalForecastTimeSeries>
	<minForecastLength multiplier="3" unit="hour"/>
	<maxForecastLength multiplier="30" unit="hour"/>
</forecastLengthEstimator>

The forecast length is defined by the length of the external forecast time series (in this example ImportCOSMO2). You can define a mimimum and / or maximum forecast length (minForcastLength / maxForecastLength). If the actual forecast length of the external forecast looked at is shorter than the minimum forecast length the forecast length is set to this minimum length (in this example 3 hours). If the actual forecast length is longer than the maximum forecast length the forecast length is set to this maximum length (in this example 30 hours).

The logging will provide you with the information which forecast length was used in the run, see example below

[] INFO - Workflow.run - Workflow.ActivityStarted: Workflow 'HBV_FlowForecast_ECMWF'
[] INFO - WorkflowPluginActivity.run - Started Activity ForecastLength_ECMWF
[] INFO - ForecastLengthEstimator.updateForecastLength - ForecastLengthEstimator.Info: Established Forecast Length as 234 hours

Here is a configuration example of the forecast length estimator using a simulated forecast time series. Make sure that your are using the read write mode "read complete forecast" or define a relative view period instead.

<?xml version="1.0" encoding="UTF-8"?>
<forecastLengthEstimator xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd">
	<simulatedForecastTimeSeries>
			<moduleInstanceId>HBV_AlpenRhein_COSMO2</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Q.fh</parameterId>
			<locationId>H-2473</locationId>
			<timeSeriesType>simulated forecasting</timeSeriesType>
			<timeStep unit="hour"/>
			<readWriteMode>read complete forecast</readWriteMode>
	</simulatedForecastTimeSeries>
	<minForecastLength multiplier="3" unit="hour"/>
	<maxForecastLength multiplier="30" unit="hour"/>
</forecastLengthEstimator>

The endoverrulable attribute in the relative view period in time series sets must be set to true in all subsequent modules in which you want to use the actual forecast length.

Since 2011.02, the forecastLengthEstimator can also be used to set a fixed end date, e.g. the end of the water year.

<?xml version="1.0" encoding="UTF-8"?>
<forecastLengthEstimator xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd">
 <timeZone>GMT-12:00</timeZone>
 <externalForecastTimeSeries>
 	<moduleInstanceId>ImportDATACARD</moduleInstanceId>
 	<valueType>scalar</valueType>
 	<parameterId>MAP</parameterId>
 	<locationSetId>Catchments</locationSetId>
 	<timeSeriesType>external forecasting</timeSeriesType>
 	<timeStep unit="hour" multiplier="6"/>
 	<relativeViewPeriod unit="day" start="0" end="365" endOverrulable="true"/>
 	<readWriteMode>read only</readWriteMode>
 </externalForecastTimeSeries>
 <minForecastEndDay>--09-30</minForecastEndDay>

</forecastLengthEstimator>

As of 2012.02, the forecast length can be forced to the end of the 2nd wateryear by using a combination of <minForecastLength> and <minForecastEndDay>. The code will first evaluate the minForecastLength before evaluating <minForecastEndDay>.

	<minForecastLength multiplier="365" unit="day"/>
	<minForecastEndDay>--09-30</minForecastEndDay>

requiredExternalForecastTimeSeries

External forecast time series set that must be available in order to run. When not available the run will be skipped. These forecast time series do not affect the T0 of the run.

This can be useful in combination with the time series set element <externalForecastMaxAge> or <relativeForecastSearchPeriod> to make sure there is a forecast available that is not to old.

In the example below a requiredExternalForecastTimeSeries is configured which will prevent the run from executing if there is no forecast available for it that is 1 day or less old.

	<requiredExternalForecastTimeSeries>
		<moduleInstanceId>NotUsedForT0_ButNeedsToBePresent</moduleInstanceId>
		<valueType>scalar</valueType>
		<parameterId>par</parameterId>
		<locationId>loc</locationId>
		<timeSeriesType>external forecasting</timeSeriesType>
		<timeStep unit="hour"/>
		<externalForecastMaxAge unit="day" multiplier="1"/>
		<readWriteMode>read complete forecast</readWriteMode>
	</requiredExternalForecastTimeSeries>

Simulated historical timeseries

This module can also be used to determine the start of the period by finding the last (approved) warm state of a specific moduleinstanceId.

<forecastLengthEstimator xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd">
    <stateModuleInstanceId>wflow_update_run</stateModuleInstanceId>
    <stateSelection>
        <stateSearchPeriod unit="hour" start="-240" end="-1"/>
    </stateSelection>
</forecastLengthEstimator>

The startoverrulable attribute in the relative view period in time series sets must be set to true in all subsequent modules in which you want to use the actual forecast length. Also make sure the statesearchperiod is the same as defined in the ForecastLengthEstimator.

External historical time series

In contrary to external forecast and simulated forcast time series, external historical time series can not be used to determine the forecast length because they are not forecasts. These time series can be configured but will only be used to determine the last time step of measuments and use that time to set the last observation time which in turn can be used to export to a run file or to set the t0 to the latest available time step.

<?xml version="1.0" encoding="UTF-8"?>
<forecastLengthEstimator xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
						 xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd">
	<externalHistoricalTimeSeries>
		<moduleInstanceId>test</moduleInstanceId>
		<valueType>scalar</valueType>
		<parameterId>par</parameterId>
		<locationId>loc</locationId>
		<timeSeriesType>external historical</timeSeriesType>
		<timeStep unit="hour"/>
		<relativeViewPeriod unit="hour" start="-5" end="0"/>
		<readWriteMode>read only</readWriteMode>
	</externalHistoricalTimeSeries>
	<setTime0ToLatestNonMissing>true</setTime0ToLatestNonMissing>
</forecastLengthEstimator>

findLatestCommonExternalForecastTime

When your forecast depends on multiple external forecasts (e.g. of an NWP forecast and a Hydro forecast) you may want to harmonize that external forecast time. This can be done by identifying the common external forecast time in the ForecastLengthEstimator and using that as a identifier in the timeseriesset used in the workflow. This identifier determines the external forecast for read and write from and to a time series set.

<?xml version="1.0" encoding="UTF-8"?>
<forecastLengthEstimator xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
						 xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd">
	<findLatestCommonExternalForecastTime externalForecastTimeId="test">
		<eventCodeOnChange>forecast.newReference</eventCodeOnChange>
		<dataFeed id="dataFeed1">
			<timeSeriesSet>
				<moduleInstanceId>test1</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>par</parameterId>
				<locationId>loc</locationId>
				<timeSeriesType>external forecasting</timeSeriesType>
				<timeStep unit="hour"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</dataFeed>
		<dataFeed id="dataFeed2">
			<timeSeriesSet>
				<moduleInstanceId>test2</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>par</parameterId>
				<locationId>loc</locationId>
				<timeSeriesType>external forecasting</timeSeriesType>
				<timeStep unit="hour"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</dataFeed>
	</findLatestCommonExternalForecastTime>
</forecastLengthEstimator>

<timeSeriesSet>
	<moduleInstanceId>ImportTMSI</moduleInstanceId>
	<valueType>scalar</valueType>
	<parameterId>P</parameterId>
	<locationId>Rain93</locationId>
	<timeSeriesType>external forecasting</timeSeriesType>
	<timeStep unit="minute" multiplier="10"/>
	<externalForecastTimeId>test</externalForecastTimeId>
	<readWriteMode>read complete forecast</readWriteMode>
</timeSeriesSet>

<general>	
	<exportType>PI 1.5</exportType>
	<folder>../junit_test_output/nl/wldelft/fews/system/plugin/dataExport/TimeSeriesExportTest/exportPiExternalForecastTimeId/export</folder>
	<exportFileName>
		<name>rainfallPiExternalForecastTimeId.xml</name>
	</exportFileName>
	<idMapId>Telemetry</idMapId>
	<unitConversionsId>UnitConversions</unitConversionsId>
	<flagConversionsId>FlagConversions</flagConversionsId>
	<exportMissingValueString>-999.0</exportMissingValueString>
	<externalForecastTimeId>test</externalForecastTimeId>
</general>

time0CardinalTimeStep

Time step which the calculated time 0 should match. When this does not match the run is skipped with status set to redundant.

skipRunWhenTime0EqualToLastRun

When this option is true, after the T0 is determined, it is compared to the T0 of the previous run and if it is equal the run is skipped with status set to redundant.

This can be useful when the T0 is determined by the oldest of multiple different forecast time series which are all created by other workflows which trigger the execution of this workflow.

This way the real execution of the workflow will only happen when all forecasts are new compared to the previous run.

Sample input and output

Sample input and output

Error and warning messages

Description of errors and warnings that may be generated

Error:	Error message
Action:	Action to fix

Known issues

Before the 2023.01 a config error will be logged when used inside a parallel block in the workflow.

Technical reference

Link to schema:	https://fewsdocs.deltares.nl/schemas/version1.0/forecastLengthEstimator.xsd

Contents