Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


Within Delft-FEWS, the datasets are exported to the archive in a workflow using the ExportArchiveModule. This workflow needs to be scheduled on a regular interval to archive all relevant data.Simulated data sets are preferably exported to archive from the workflow which created the simulation. This is especially the case when modifiers are used in the simulation. When a simulation is not exported to the archive directly it might be that some modifiers which are used in the simulation are deleted are changed in between. This will cause that it won't be possible to archived all the used modifiers correctly.

 

To prevent dependencies on other processes, the ExportArchiveModule is envisioned to write directly into the archive file storage. The FSS thus needs to be able to have write access to those disks.
For each kind of dataset, the ExportArchiveModule checks the database for changes over a (configured) relative period. It exports any data which meets the export instructions and has changed within this period. Datasets are archived in a pre-defined directory structure, which is based on areaId, date and dataset.


The schema of the associated configuration file (Figure 4.1) is defined at:
http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd.

Figure 4.1 Top level of Delft-FEWS exportArchiveModule.xsd

Table of Contents
maxLevel1

Observations archiving by Delft-FEWS

...

*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.


Figure 4.2 Deft-FEWS export configuration for archiving observations

Amalgamate observations

With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration.

Element

Format

Description

GeneralAmalgamateObservedSection ComplexType

 

 

archiveFolder

string

Export destination folder, assumes that the account running the FEWS (FSS) application has write access

relativePeriod

 

period experiencing changed data that should be amalgamted

AmalgamateActivities ComplexType

 

 

amalgamateObservedData

AmalgamateActivity

indicator that observed data will be amalgamated

areaId

string

area for which amalgamation takes place


In future, the amalgamate process will be executed on the archive serverside via the ArchiveAdminConsole.


Anchor
_Toc386122334
_Toc386122334
Messages archiving by Delft-FEWS

...