One of the most important concepts of Delft-FEWS is that always the latest forecast data is available to be used. In a typical operational Delft-FEWS forecasting system, new data is constantly imported, while older data is no longer relevant and can be removed. For a lot of database tables in the central Delft-FEWS database, the rows (records) in the database have an expiryTime column specifying when the data can be removed. This expiryTime is set at creationTime. The removal is done by a continous running process in the Master Controller called the McRollingBarrel. The McRollingBarrel - in 10 minute intervals - removes data for which the expiry time is exceeded and no dependent rows are available. This is often critical for maintaining an optimal application performance.
The global property DEFAULT_EXPIRY_DAYS (default 10 days) is a global property that controls the default expiration time for all forecast data that is stored in the Delft-FEWS database.
This will override the expiryTime in days when log entries created by the client will expire.
This will override the expiryTime in days when log entries created by the client will expire.
Delft-FEWS allows for fine grained control of expiry times for timeseries by specifying the default in a global propery and overriding the expiry time on a deeper level for instance in the configuration of the import. Once configured, it is good practive to verify the expiry times of imported data using Time Series Lister.
The System default expiry time for forecast data in Delft-FEWS is 10 days. The system default can be altered by specifying a global property, e.g. DEFAULT_EXPIRY_DAYS=30
DEFAULT_EXPIRY_DAYS=30 |
Import workflows can have expiry time in general section, will be used for all imported time series
<expiryTime unit="day" multiplier="365"/> |
Import, Transformation, General Adapter, .. modules can have expiry times in timeSeriesSet element, e.g.
<expiryTime unit="day" multiplier="2"/> |
The expiryTime value in the TimeSeriesSet configuration of a TimeSeriesImport overrides DEFAULT_EXPIRY_DAYS setting for the expiry time of TimeSeries and ImportStatus. If an expiryTime is defined at the TimeSeriesSet level, the time series blobs will get this expiry time, no matter what the workflow says.
If the timeseries is of an external type, the timeseries will also stay accessible. If a time series is of the simulated type, data access has to go through a query where the taskrun record plays a major role. This taskrun record expirytime is defined by the workflow.
When expiryTime of many TimeSeriesSets are conceptually related, it is recommended to explicitely set these using a globalProperty variable using tags. |
Set the expiry times in the configuration as brief as possible in order to keep the database tables as small as possible and performing well. If needed use the Open Archive to store data for longer. When the Delft-FEWS is used as a forecasting system it is common to use 2, 7 or 10 days whereas Water Information System uses much longer periods. Expiry time should be what is required for the daily work, e.g.
NB. Another means of For intermediate data processing steps - that do not need to be displayed - the data does not even need to used in displays, use temporary series (synchlevel 9)
This setting overrides the expiry time of the taskrun and everything attached to it. The ‘runExpiryTime’ applies only to forecast runs, so only to time series created in a forecast run. Imported time series (external historicals or external forecasts) are not connected to a particular workflow run, so ‘runExpiryTime’ will not work for imported timeseries.
The expiryTime value of thresholdEvents is the maximum of 365 days and the DEFAULT_EXPIRY_DAYS setting. There is also a eventExpiryTime in the ThresholdGroups.xml
Forecaster notes have a default expiry time of 5 days. This can be overruled per event code in the ForecasterNotesDisplay.xml file.
<eventCode id="Comms"><expiryTime unit="day" multiplier="10"/></eventCode> |