Introduction on Expiry time
One of the most important concepts of Delft-FEWS is that always the latest forecast data is available to be used. In a typical operational Delft-FEWS forecasting system, new data is constantly imported, while older data is no longer relevant and can be removed. For a lot of database tables in the central Delft-FEWS database, the rows (records) in the database have an expiryTime column specifying when the data can be removed. The removal is done by the so-called RollingBarrel process. It will remove any data for which the expiry time is exceeded.
GlobalProperty DEFAULT_EXPIRY_DAYS
The global property DEFAULT_EXPIRY_DAYS (default 10 days) is a global property that controls the default expiration time for new forecast data that is stored in the Delft-FEWS database.
GlobalProperty DEFAULT_EXPIRY_DAYS_LOGEVENT
This will override the expiryTime in days when log entries created by the client will expire.
GlobalProperty DEFAULT_EXPIRY_DAYS_LOGEVENT_MANUAL
This will override the expiryTime in days when log entries created by the client will expire.
WorkflowDescriptors runExpiryTime
This setting overrides the expiry time of the taskrun and everything attached to it. The ‘runExpiryTime’ applies only to forecast runs, so only to time series created in a forecast run.
Imported time series (external historicals or external forecasts) are not connected to a particular workflow run, so ‘runExpiryTime’ will not work for imported timeseries.
Expiry time for TimeSeries and ImportStatus
The expiryTime value in the TimeSeriesSet configuration of a TimeSeriesImport overrides DEFAULT_EXPIRY_DAYS setting for the expiry time of TimeSeries and ImportStatus. If an expiryTime is defined at the TimeSeriesSet level, the time series blobs will get this expiry time, no matter what the workflow says.
If the timeseries is of an external type, the timeseries will also stay accessible. If a time series is of the simulated type, data access has to go through a query where the taskrun record plays a major role. This taskrun record expirytime is defined by the workflow. If the taskrun record is removed, the simulated data has become orphan as it has become inaccessible.
Use globalProperties for controlling expiryTime timeSeriesSets...
When expiryTime of many TimeSeriesSets are conceptually related, it is recommended to explicitely set these using a globalProperty variable using tags.
Preserving taskrun information for access to timeseries (before 2017.01)
Timeseries in the database without taskruns are referred to as "orphaned timeseries". Since 2017.01, taskruns for new timeseries are preserved until the timeseries expire, and therefore the newly created timeseries cannot become orphaned anymore. Note that simulated timeseries are not meaningful without taskrun information and should never be orphaned. Only external timeseries can be orphaned, but without taskruns they can not be amalgamated (merged) any more. In order to preserve taskruns longer than the default of ten days, the taskruns expirytime should be increased. See also ImportAmalgamate for more information on amalgamate.
Expiry time for thresholdEvents
The expiryTime value of thresholdEvents is the maximum of 365 days and the DEFAULT_EXPIRY_DAYS setting. There is also a eventExpiryTime in the ThresholdGroups.xml
Expiry time for Forecaster Notes
Forecaster notes have a default expiry time of 5 days. This can be overrules on event code in the ForecasterNotesDisplay.xml file.
<eventCode id="Comms"><expiryTime unit="day" multiplier="10"/></eventCode>
MCRolling barrel extends expiry times for SystemActivities, Tasks and TaskRuns when necessary (2017.01 and later)
The MC RollingBarrel component will extend the expiryTime of system activities and tasks when the rows are almost expired and there is still forecast data attached.
Expiry time for records generated by the master-controller (2017.01 and before)
The master-controller configuration defines the following section section in the fews.master.mc.conf configuration file.
<rollingbarrel>
<taskruns expiredays="10"/>
<logentries expiredays="5"/>
<reports expiredays="5"/>
<default expiredays="10"/>
</rollingbarrel>
These settings are used by master-controller components when creating records. The taskruns expiry time used is read from the taskproperties, when missing the above setting is used.
An edit of a task in the Admin Interface will update the taskproperties.xml to the value in <taskruns expiredays="..."/> setting.
