Introduction on Expiry time
One of the most important concepts of Delft-FEWS is that always the latest forecast data is available to be used. In a typical operational Delft-FEWS forecasting system, new data is constantly imported, while older data is no longer relevant and can be removed. For a lot of database tables in fews master-controller database, the rows (records) in the database have an expiryTime column specifying when the data can be removed. The removal is done by the so-called RollingBarrel process. It will remove any data for which the expiry time is exceeded.
GlobalProperty DEFAULT_EXPIRY_DAYS
The global property DEFAULT_EXPIRY_DAYS (default 10 days) is a global property that controls the default expiration time for new forecast data that is stored in the Delft-FEWS database.
GlobalProperty DEFAULT_EXPIRY_DAYS_LOGEVENT
This will override the expiryTime in days when log entries created by the client will expire.
GlobalProperty DEFAULT_EXPIRY_DAYS_LOGEVENT_MANUAL
This will override the expiryTime in days when log entries created by the client will expire.
WorkflowDescriptors runExpiryTime
This setting overrides the expiry time of the taskrun and everything attached to it. The ‘runExpiryTime’ applies only to forecast runs, so only to time series created in a forecast run.
Imported time series (external historicals or external forecasts) are not connected to a particular workflow run, so ‘runExpiryTime’ will not work for imported timeseries.
Expiry time for TimeSeries and ImportStatus
The expiryTime value in the TimeSeriesSet configuration of a TimeSeriesImport overrides DEFAULT_EXPIRY_DAYS setting for the expiry time of TimeSeries and ImportStatus.
If an expiryTime is defined at the TimeSeriesSet level, the time series blobs will get this expiry time, no matter what the workflow says.
If the timeseries is of an external type, the timeseries will also stay accessible. If a time series is of the simulated type, data access has to go through a query where the taskrun record plays a major role. This taskrun record expirytime is defined by the workflow. If the taskrun record is removed, the simulated data has become orphan as it has become inaccessible.
Preserving taskrun information
Timeseries in the database without taskruns are referred to as "orphaned timeseries". Note that simulated timeseries are not meaningful without taskrun information and should never be orphaned. Only external timeseries can be orphaned, but without taskruns they can not be amalgamated (merged) any more. In order to preserve taskruns longer than the default, the taskruns expirytime should be increased.
Expiry time for thresholdEvents
The expiryTime value of thresholdEvents is the maximum of 365 days and the DEFAULT_EXPIRY_DAYS setting.
Expiry time for records generated by the master-controller
The master-controller configuration defines the following section section in the fews.master.mc.conf configuration file.
<rollingbarrel>
<taskruns expiredays="10"/>
<logentries expiredays="5"/>
<reports expiredays="5"/>
<default expiredays="10"/>
</rollingbarrel>
These settings are used by master-controller components when creating records. The taskruns expiry time used is read from the taskproperties, when missing the above setting is used.
An edit of a task in the Admin Interface will update the taskproperties.xml to the value in <taskruns expiredays="..."/> setting.
