Page History

...

...

The DELFT-FEWS databases (central database and local data store) are not meant to store forecast related data indefinitely. To make sure that the database system does not fill up, a rolling barrel of configurable length is applied to all data. All dynamic data is kept in the system for a configurable length of time. This is identified by setting an expiry time to each item of dynamic data saved in the database. DELFT-FEWS can run a module to remove data for which the expiry date has passed from the database. This module should be scheduled to run on a daily basis on the live system (usually at times when the system is not used extensively- e.g. around midnight). On operator clients the rolling barrel is run when the system identifies that enough data is expired to make the run worthwhile.

As all functional tasks are run by DELFT-FEWS through a workflow, a moduleInstance is created to allow the rolling barrel to be run. The module instance must be correctly registered in the moduleInstanceDescriptors (see Regional Configuration), and point to the relevant class in the moduleDesriptors configuration (see System Configuration). The module does not require any configuration. There is therefore not an XML file available, nor need one be configured to run this module.

Anchor
_Toc154574464 _Toc154574464

Compact index and cache files

Because time series are combined in one record, FEWS uses a custom index to know which records are involved when reading time series. Expired records are automatically removed from the database but not from the index files that reside on forecasting shells. To do this the compact cache files activity should be scheduled on every forecasting shell daily. On operator clients and stand-alone systems this happens automatically. When the obsolete ModuleDescriptors.xml still exist the module “nl.wldelft.fews.system.plugin.rollingbarrel.RollingBarrelModule” is redirected to the compact cache file activity. The compact cache file workflow is not running the rolling barrel any more. The rows deleted by the MC Rolling Barrel are removed every time the forecasting shell starts when a local data store is used.

Compact_FSSCache.xml in WorkflowFiles

<?xml version="1.0" encoding="UTF-8"?>
<workflow version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.wldelft.nl/schemas/version1.0/workflow.xsd">
	<activity>
		<predefinedActivity>compact cache files</predefinedActivity>
	</activity>
</workflow>

WorkflowDescriptors.xml in RegionConfigFiles

<workflowDescriptor id="Compact_FSSCache" name="Compact FSS Cache Files" forecast="false" visible="true">
	<runExpiryTime unit="week" multiplier="52"/>
</workflowDescriptor>

In the workflow mapping, the workflow should be set to; staggered-all. This will make sure the workflow is run on all FSS's, but not at the same time. Staggering time should be minimal 15 minutes.

Example of a log message when the compact cache is finished:
20-02-2016 04:53:46 INFO - TimeSeriesIndex.CompactFinished: 6 seconds, removed rows 142844/403158, removed time series 923/50598, removed groups 39168/100775, removed ids 8/13565, removed qualifier sets 0/0, removed id sequences 27256/78750, released memory 10 MB/45 MB

Every unique combination of time series that shares one record is called a group. Every unique sorted combination of ids for parameters, locations and qualifiers that exist in any group is called an id sequence. A unique sorted combination of qualifiers attached to one or more time series is called a qualifier set. Multiple qualifiers can be attached to a single time series.

Running the compact cache files regularly not only reduces disk space but also make the indexes more memory efficient and faster.
When an external state zip directory is configured in the clientConfig.xml this module is also deleting state zip files that no longer exist in the database.Image Removed