How to influence the expiry time of rows (records) in the central database

Introduction to expiryTime

One of the most important concepts of Delft-FEWS is that always the latest forecast data is available to be used. In a typical operational Delft-FEWS forecasting system, new data is constantly imported, while older data is no longer relevant and can be removed. For a lot of database tables in the central Delft-FEWS database, the rows (records) in the database have an expiryTime column specifying when the data can be removed. This expiryTime is set at creationTime.
The removal is done by a continuous running process in the Master Controller called the McRollingBarrel. The McRollingBarrel - in 10 minute intervals - removes data for which the expiry time is exceeded and no dependent rows are available. This is often critical for maintaining an optimal application performance.

How to set the expiryTime

global.properties

There are default properties that can be defined through the global.properties file.

• DEFAULT_EXPIRY_DAYS:  (default 10 days) controls the default expiration time for all forecast data that is stored in the Delft-FEWS database.
• DEFAULT_EXPIRY_DAYS_LOGEVENT: (default 5 days) controls the default expiration time for non-manual log events that are stored in the Delft-FEWS database.
• DEFAULT_EXPIRY_DAYS_LOGEVENT_MANUAL: (default 5 days) controls the default expiration time manual log events that are stored in the Delft-FEWS database.

WorkflowDescriptors runExpiryTime

This setting overrides the expiry time of the taskrun and everything attached to it.
Used for runs of simulated timeseries to override expiry time from global properties DEFAULT_EXPIRY_DAYS.
When timeseriesSet expiryTime is also configured, the minimum of runExpiryTime and the timeSeriesSets expiryTime is used.

• The ‘runExpiryTime’ applies only to forecast runs, so only to time series created in a forecast run. 
  
• Imported time series (external historicals or external forecasts) are not connected to a particular workflow run, so ‘runExpiryTime’ will not work for imported timeseries.

timeSeriesSet expiryTime in Modules

Delft-FEWS allows for fine grained control of expiry times for timeseries by specifying the default in a global property and overriding the expiry time on a deeper level for instance in the configuration of the import. 
Once configured, it is good practice to verify the expiry times of imported data using Time Series Lister.

The expiryTime value in the TimeSeriesSet configuration of a TimeSeriesImport overrides DEFAULT_EXPIRY_DAYS setting for the expiry time of TimeSeries and ImportStatus.
If an expiryTime is defined at the TimeSeriesSet level, the time series blobs will get this expiry time, no matter what the workflow says.

• If the timeseries is of an external type, the timeseries will also stay accessible. 
• If the time series is of a simulated type, data access has to go through a query where the taskrun record plays a major role. This taskrun record expiryTime is defined by the workflow.

thresholdEvents eventExpiryTime

The expiryTime value of thresholdEvents is the maximum of 365 days and the DEFAULT_EXPIRY_DAYS setting.
This can be overruled by the configuration option <eventExpiryTime> in the ThresholdGroups.xml

Forecaster Notes expiryTime

Forecaster notes have a default expiry time of 5 days.
This can be overruled per event code in the ForecasterNotesDisplay.xml file. 

<eventCode id="Comms"><expiryTime unit="day" multiplier="10"/></eventCode>

Hierarchy of expiryTime settings

1. System default expiry time for forecast data in Delft-FEWS is 10 days.

2. Global Properties can overrule the system default, e.g. DEFAULT_EXPIRY_DAYS=30

   DEFAULT_EXPIRY_DAYS=30

3. Workflows for importing series can have expiry time in general section, which will be used for all imported time series

   <expiryTime unit="day" multiplier="365"/>

   1. Workflows that are Scheduled as Tasks in the Admin Interface can have expiry time defined in Task Run Expiry Time. 
       Default is to take the runExpiry time of the workflow. If this is not defined, the expiry time configured on the MC is the default.

4. Modules such as Import, Transformation, General Adapter, .. can have expiry times in timeSeriesSet element, e.g.

   <expiryTime unit="day" multiplier="2"/>

Guidelines for setting expiry times

Set the expiry times in the configuration as brief as possible in order to keep the database tables as small as possible and performing well. Expiry time should be what is required for the daily work, e.g.
When the Delft-FEWS is used as a forecasting system it is common to use 2, 7 or 10 days whereas Water Information System uses much longer periods. 
Observed and processed (external historical) time series, 2-6 months (>1 year for Astro)

1. 1. Historic simulations (simulated historical) time series and model states, 10 days to 6 months
   2. Imported forecasts (external forecasts) time series, 2 hours to 7 days
   3. Forecast simulations (simulated forecasts) time series, 2 to 10 days
   4. Intermediate data - e.g. from processing steps - that do not need to be displayed, use temporary series (synchlevel 9)

The Open Archive can be used to store data for longer periods.