...
Within Delft-FEWS, the datasets are exported to the archive in a workflow using the ExportArchiveModule. This workflow needs to be scheduled on a regular interval to archive all relevant data.Simulated data sets are preferably exported to archive from the workflow which created the simulation. This is especially the case when modifiers are used in the simulation. When a simulation is not exported to the archive directly it might be that some modifiers which are used in the simulation are deleted are changed in between. This will cause that it won't be possible to archived all the used modifiers correctly.
To prevent dependencies on other processes, the ExportArchiveModule is envisioned to write directly into the archive file storage. The FSS thus needs to be able to have write access to those disks.
For each kind of dataset, the ExportArchiveModule checks the database for changes over a (configured) relative period. It exports any data which meets the export instructions and has changed within this period. Datasets are archived in a pre-defined directory structure, which is based on areaId, date and dataset.
The schema of the associated configuration file (Figure 4.1) is defined at:
http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd.
Figure 4.1 Top level of Delft-FEWS exportArchiveModule.xsd
| Table of Contents | ||
|---|---|---|
|
Observations archiving by Delft-FEWS
...
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.
Figure 4.2 Deft-FEWS export configuration for archiving observations
Amalgamate observations
With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration.
Element | Format | Description |
GeneralAmalgamateObservedSection ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| period experiencing changed data that should be amalgamted |
AmalgamateActivities ComplexType |
|
|
amalgamateObservedData | AmalgamateActivity | indicator that observed data will be amalgamated |
areaId | string | area for which amalgamation takes place |
In future, the amalgamate process will be executed on the archive serverside via the ArchiveAdminConsole.
| Anchor | ||||
|---|---|---|---|---|
|
...