Page History

...

One of the most important properties of DELFT Delft-FEWS as a forecasting system is its ability to efficiently deal with large volumes of dynamic data.  Dynamic Dynamic data covers mainly time series data in various formats (scalar- 0D, vector - 1D, longitudinal profile - 1D and 2D, grid - 2D and 3D, and polygon data - 2D). Dynamic data also includes the management of model states produced by the system. As Delft-FEWS is not only used as operational forecasting system, but also as a system to run climate scenario's scenarios, the length of each time series can be up up to 2,000,000,000 time steps. A thorough understanding of how DELFTDelft-FEWS handles dynamic data is fundamental in the correct configuration of to correctly configuring an operational system. For Specific optimisations are available for each type of the different types of dynamic data specific optimisations have been introduced.To allow handling of time series data, . This chapter introduces the concept of a "Time Series Set" is introduced. A Time Series Set is used to retrieve data from and submit data to the database. In this chapter the concept of the time series set is explained.

Types of time series

External and Simulated time series

Time series are considered to be available from two sources. All time series sourced from external systems are considered as "External". All time series produced by the forecasting system itself are considered as "Simulated".

Forecast and Historical time series

Time series are considered to be of two categories in relation to time. Historical time series are continuous time series that describe a parameter at a location over a period of time. Forecast time series are different to historical time series in that for each location and parameter one forecast is independent of another forecast. A forecast is characterised by its start time and the period it covers. Generally, when a new forecast is available for a given location and parameter, it will supersede any previous forecast for that location and parameter. Each forecast is therefore an independent entity.

On the basis of this, four six categories of time series are identified;

...

There are significant differences in how each of these time series are handled in Delft-FEWS.

External Historical time series

In an online operational system DELFT, Delft-FEWS will incrementally import imports observed data as it becomes available from external systems. This These data should be imported as an External Historical time series. When data marked as external historical is presented to the system with exactly the same values and covering the same period as data for that location/parameter already available in the database, then it will be ignored (i.e. not imported). Only new data is are imported and stored. If data for a given period is already available but is changed (manual edit or update), then the new values will be added to the database. For each item of data added to the database, a time stamp is included to specify when the data was made available to the system.

When data of the external historical type is data are requested from the database, the most recently available data over that whole period is returned. If the data for that period was imported piecewise, then the individual pieces will be merged prior to the data being returned. An example is given in Figure 1, where data is are imported sequentially. Each data dataset imported/edited is indicated using a different line style. At When the request for the complete series (a) is requested, the most recent data available over the complete period is merged and returned. The data imported at 12:00 partially overlaps that imported at 10:00. As Since the 12:00 data is the most recent, it will persist in the complete series. A manual edit may be done (or interpolation) may be done to fill the gap in the data. This will be returned in a subsequent request for the complete series. Although a complete series is returned, the data is stored as it is imported, including a time stamp indicating when the import happened. If at a later stage the data available at directly preceding the manual edit is requested, then the additional data will not be included in the complete series.

...

External forecasts are imported by DELFTDelft-FEWS as these they are made available by the other, external forecasting systems. Again each Each forecast is imported and stored individually. External forecasts are referenced by the start time of that the forecast. When retrieving an external forecast time series from the database, the most recent recently available forecast, as indicated by the forecast start time, will be returned. The most recently available forecast is determined as the latest forecast with a start time earlier or equal to the start of the forecast to be made using DELFTDelft-FEWS (forecast T0). It is thus not possible to see an external forecast time series on request, as the latest available is always returned.

...

Simulated historical time series are similar to the external historical time series in that they are continuous in time. The difference is that the time series are referenced through the forecast (model) run they have been produced by. As a consequence the time series can be retrieved either by directly requesting it through opening the run and viewing, or if the run is approved. If you use an extended relativeViewPeriod and the readWriteMode "read only" with a simulated historical time series, you can access the combined results of several model runs (within the specified relativeViewPeriod), similar to the default behavior of the merged external historical time series. If you use readWritemode readWriteMode 'read complete forecast' without a relativeViewperiodrelativeViewPeriod, you will only obtain the Current current forecast.

Simulated historical time series are generally produced by model runs where a model initial state is used. Each time series has a history, i.e. the state used as its initial condition. Each state again has a history, i.e. the model run that produced the state. This history is used by the database in constructing a continuous time series.

...

Figure2 schematically shows how a sequence of runs producing simulated historical and simulated forecasting time series are stored. Each simulated historical run uses the module state saved at the end of the previous run. It can be seen that these simulated historical traces are treated as a continuous time series when requested later. For the forecasting time series, only the most recent (approved) time series is displayed.

Figure2 Figure 2 Schematic overview of handling simulated forecasting and simulated historical time series. Three subsequent forecasts are shown, and the resulting complete time series returned when requested after 12:00. The historical time series is traced back using the state used to create the link to a previous run. For the forecast time series the most recent forecast supersedes previous forecasts.

...

Time Series Sets

Any module in DELFTDelft-FEWS that requires data from the database, or produces data that must be stored in the database, does so through the use of a complex data type referred to as the Time Series Set. A time series set can be compared to a query that is run against the database. It contains all the keys to uniquely identify the set of data to be retrieved (for more information on key attributes, see Key attributes).

...

Optional items may also need to be required to fulfil fulfill the requirements of the module using the time series set. This will be indicated in this manual for those modules where appropriate.

...

The relative view period defines the span of time for which data is to be retrieved. This span of time is referenced to the start time of the forecast run (T0) the time series set is used in. If the time series set is not used in a forecast run (e.g. in the displays), then the reference is to the DELFTDelft-FEWS system time.

Parameters

...

•         divider same function as the multiplier, but defines fraction of units.

•         timeZone defines the timeZone, this is only relevant for units of a day or larger.

readWriteMode

• the multiplier, but defines fraction of units.

•         timeZone defines the timeZone, this is only relevant for units of a day or larger.

externalForecastSearchTimeStep

Since 2024.01. Only time series with an external forecast time that match this time step are visible while searching, e.g. an externalForecast timeseries that has a  externalForecastTime of 18:00 in the GMT timeZone.

All timestep attributes are available, probably most relevant options are;

•        id  Id of the time step. You can reference time steps defined in the regionConfig/timeSteps.xml.    
•        times Defines the time step by a list of times without dates, e.g. "10:00 23:00"
•         timeZone defines the timeZone, this is only relevant for units of a day or larger.

<timeSeriesSet>
	<moduleInstanceId>ImportACCESS-GE</moduleInstanceId>
	<valueType>grid</valueType>
	<parameterId>P.nwp.fcst</parameterId>
	<locationId>ACCESS-GE_grid</locationId>
	<timeSeriesType>external forecasting</timeSeriesType>
	<timeStep unit="hour"/>
	<relativeForecastPeriod unit="hour" start="0" end="246"/>
	<externalForecastSearchTimeStep times="18:00" timeZone="GMT"/>
	<readWriteMode>read complete forecast</readWriteMode>
</timeSeriesSet>

readWriteMode

The readWriteMode The readWriteModel definition is mainly used in the definition of filters to be applied in the time series display when used in edit mode. This element is an enumeration of;

...

• Manually corrected (not completed) imported values will not be rollbacked rolled back when the value is reimported.
• Values added in the period after T0 in an external historical timeserietimeseries, WILL be overwritten by imported values.
• All other modules that generate the timeserie timeseries (such as a transformation)  WILL WILL overwrite manually edited data.

...