The Delft-FEWS ExportArchiveModule
Within Delft-FEWS, the datasets are exported to the archive in a workflow using the ExportArchiveModule. This workflow needs to be scheduled on a regular interval to archive all relevant data. To prevent dependencies on other processes, the ExportArchiveModule is envisioned to write directly into the archive file storage. The FSS thus needs to be able to have write access to those disks.
For each kind of dataset, the ExportArchiveModule checks the database for changes over a (configured) relative period. It exports any data which meets the export instructions and has changed within this period. Datasets are archived in a pre-defined directory structure, which is based on areaId, date and dataset.
The schema of the associated configuration file (Figure 4.1) is defined at:
http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd.
Figure 4.1 Top level of Delft-FEWS exportArchiveModule.xsd
Observations archiving by Delft-FEWS
For observations a dataset is generated for every area on a daily basis. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archive root>/<yyyy>/<MM>/<areaId>/<dd>/observed/
When Delft-FEWS generates the netcdf file, data is written to the same data block when the entire matrix is filled, i.e. all time steps are regular and none of the values is missing. For those locations with irregular time stamps, or missing values, a separate data block is used.
Within the netCDF file, each data block is accompanied by a header. Within each header a metadata item called 'timeseries_sets_xml' is included, holding the exact definition of the timeSeriesSet as the data was stored in the FEWS database. This feature allows full reproducibility of the time series via an Import workflow in a Delft-FEWS stand alone application, assuming that the associated Delft-FEWS configuration is in place.
The exportArchiveModule.xsd has a dedicated exportObserved section to configure the observed timeseries that need to be archived (see Figure 4.2). Table 4.1 documents the associated elements:
Table 4.1 Deft-FEWS export configuration for archiving observations
Element | Format | Description |
general ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| Exports entire dataset by day, for any day where a database change (blob creation time) is detected within the relativePeriod (relative to T0). |
idMap | string | idMap applied to translate internal FEWS identifiers to identifiers that meet NetCDF-CF criteria.E.g. netcdf does not allow a full stop ('.') in the variable name |
netcdfObservedExportActivities ComplexType |
|
|
fileName | string | without nc extension, preferably no spaces |
areaId | string | area to which the dataset belongs |
ncMetaData | string elem. | optional metadata tags within NetCDF file following CF convention. Supported by the internal catalogue of the THREDDS Data Server |
includeFlags | bool | default=TRUE; if TRUE, a list of flags is stored, each value pointing to the associated flag |
includeComments | bool | default=TRUE; if TRUE, a list of comments is stored, each value pointing to the associated comment |
thresholdGroupId | string | identifies FEWS ThresholdGroup which is used to detect threshold crossings to be highlighted in the metaData.xml |
timeSeriesSets |
| FEWS timeseries sets |
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.
Figure 4.2 Deft-FEWS export configuration for archiving observations
Messages archiving by Delft-FEWS
Delft-FEWS can export messages to the archive via the ArchiveExportModule (exportMessages activity).
For messages a dataset is generated for every area on a daily basis. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows TO DO: FEWS-10198: update directory structure to above setup:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd>/messages/
The exportArchiveModule.xsd has a dedicated exportMessages section to configure the messages that need to be archived (see Figure 4.3). Currently only ForecasterNotes can be archived. Table 4.2 documents the associated elements:
Table 4.2 Delft-FEWS export configuration for archiving messages
Element | Format | Description |
general ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| Exports entire dataset by day, for any day where a database change (blob creation time) is detected within the relativePeriod (relative to T0). Existing files are overwritten* |
MessagesActivities ComplexType |
|
|
forecasterNotesExportActivity ComplexType* |
|
|
areaId | string | area for which messages need to be archived |
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.
Figure 4.3 Delft-FEWS export configuration for archiving messages
External forecast archiving by Delft-FEWS
Delft-FEWS can export external forecast time series to the archive via the ArchiveExportModule (exportExternalForecast activity).
For external forecasts a dataset is generated for every area for every source for every forecast. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd> /external_forecasts/<sourceId>_<ExtForecastTime>
The exportArchiveModule.xsd has a dedicated exportExternalForecast section to configure the time series that need to be archived (see Figure 4.4). Table 4.3 documents the associated elements:
Figure 4.4 Delft-FEWS export configuration for archiving external forecasts
Table 4.3 Delft-FEWS export configuration for archiving external forecasts
Element | Format | Description |
GeneralExportForecastSection ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| Exports entire external forecast dataset by source and forecast time, for any forecast where a database change (blob creation time) is detected within the relativePeriod (relative to T0). Existing timeseries files are overwritten* |
idMap | string | idMap applied to translate internal FEWS identifiers to identifiers that meet NetCDF-CF criteria.E.g. netcdf does not allow a full stop ('.') in the variable name |
ExternalForecastActivities ComplexType |
|
|
NetcdfExternalForecast ExportActivities ComplexType |
|
|
NetcdfExternalForecast ExportActivity ComplexType* |
|
|
fileName | string | without nc extension, preferably no spaces |
areaId | string | area to which the dataset belongs |
sourceId |
| identifies underlying source e.g. NWP product |
ncMetaData | string elem. | optional metadata tags within NetCDF file following CF convention. Supported by the internal catalogue of the THREDDS Data Server |
includeFlags | bool | only applied for scalar values. |
includeComments | bool | only applied for scalar values. |
timeSeriesSets |
| FEWS timeseries sets |
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.
Simulations archiving by Delft-FEWS
Delft-FEWS can export simulations to the archive via the ArchiveExportModule (exportSimulations activity). Typically this archiving activity is included in the workflow that computes the final approved forecast.
The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd>/simulated/<workflowId><TimeZero><DispatchTime>
Where <dd> refers to the date of the forecast time T0.
This directory holds the metaData.xml as well as runInfo.xml file with the FEWS taskrun properties (see Figure 3.8). Within this directory the following sub-folders may exist:
/timeseries /reports /modifiers/states
The exportArchiveModule.xsd has a dedicated exportSimulated section to configure the messages that need to be archived (see Figure 3.7). The associated specification is given in Table 4.4.
Table 4.4 Delft-FEWS export configuration for archiving simulations
Element | Format | Description |
GeneralExportForecastSection ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| Exports entire simulated timeseries by workflow, for any simulated forecast where database change (blob creation time) are detected within the relativePeriod (relative to T0). Existing timeseries files are overwritten* |
idMap | string | idMap applied to translate internal FEWS identifiers to identifiers that meet NetCDF-CF criteria.E.g. netcdf does not allow a full stop ('.') in the variable name |
ForecastActivities ComplexType |
|
|
NetcdfForecastExportActivities ComplexType |
|
|
NetcdfForecastExportActivity ComplexType* |
|
|
fileName | string | without nc extension, preferably no spaces |
areaId | string | area to which the dataset belongs |
ncMetaData | string elem. | optional metadata tags within NetCDF file following CF convention. Supported by the internal catalogue of the THREDDS Data Server |
includeFlags | bool | only applied for scalar values. |
includeComments | bool | only applied for scalar values. |
timeSeriesSets |
| FEWS timeseries sets |
ReportsExportActivity ComplexType* |
|
|
subfolder | string | Subdirectory within the 'reports' directory |
moduleInstanceId | string | moduleInstanceId which created the report |
ModuleStatesExportActivity ComplexType* |
|
|
moduleInstanceId | string | moduleInstanceId which created the state |
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.
Figure 4.5 Delft-FEWS export configuration for archiving simulations
Delft-FEWS export of configuration
Delft-FEWS can export the current configuration to the archive via the ArchiveExportModule (exportConfig activity). The configuration thus is exported as part of the workflow. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<config>//<areaId>/<yyyymmdd>/
The date refers to the revision date of the configuration. The file name typically holds the revisionId.
The exportArchiveModule.xsd has a dedicated exportConfig section to setup the export of the Configuration (see Figure 4.6). This is due to be revised as the same relativePeriod based export mechanism should be adopted for checking what to export (see Table 4.5).
Figure 4.6 Delft-FEWS export configuration for archiving configurations
Table 4.5 Delft-FEWS export configuration for archiving a configuration
Element | Format | Description |
GeneralExportConfig ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod (TO DO) |
| Exports entire configuration when database change (config revision) has been detected within the relativePeriod (relative to T0). Existing files are overwritten* |
ExportConfigActivities ComplexType |
|
|
ExportConfigActivity ComplexType* |
|
|
areaId | string | area to which the dataset belongs |
Delft-FEWS export of rating curves
Delft-FEWS can export rating curves to the archive via the ArchiveExportModule (exportRatingCurves activity). The entire history of the rating curves is exported. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows FEWS-10198: update directory structure to above setup:
<archiveRoot>/ratingcurves/<areaId>
The configurator can choose between exporting the full set or just the rating curves that have changed, with or without modifier changes.
The exportArchiveModule.xsd has a dedicated exportRatingCurve section to setup the export of the Configuration. Table 4.6 and Figure 4.7 discuss the general section.
Table 4.6 Delft-FEWS export configuration for archiving rating curves (general section)
Element | Format | Description |
GeneralExportRatingCurves ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| If defined, only rating curves which have database changes during the relativePeriod (relative to T0) are exported. If no relativePeriod is specified all available rating curves are exported |
Prefix-timeZeroFormattingString | string | Adds T0 stamp as prefix to file name E.g.yyyymmdd_hhmm |
idMap | string | idMap applied to translate internal FEWS identifiers to external identifiers |
unitConversionsId | string | Allows conversion of rating table units |
includeModifiers | bool | Default=FALSE; If TRUE, rating curve modifiers should be included in the export |
Figure 4.7 Delft-FEWS export configuration for archiving rating curves, general section
Table 4.7and Figure 4.8 discuss the activities section of for exporting rating curves.
Table 4.7 Delft-FEWS export configuration for archiving rating curves (activities section)
Element | Format | Description |
RatingCurvesActivities ComplexType |
|
|
RatingCurvesExportActivity ComplexType |
|
|
fileName | string | without nc extension, preferably no spaces |
areaId | string | area to which the dataset belongs |
linearTableStageresolution | string | Stage increment between the rows in a rating table |
locationId/locationSetId | string | Locations for which rating curves should be exported to this file |
Figure 4.8 Delft-FEWS export configuration for archiving rating curves, activities section
Archiving Delft-FEWS snap shots
Snap shots of the Delft-FEWS database can be archived via the archiveExportModule (exportSnapShot activity).
For snapshots a dataset is generated and stored under the configured area at the memoment of workflow execution. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId/<snapshot>/
Figure 4.9 and Table 4.8 documents the associated elements in the exportSnapShot activity:
Table 4.8 Delft-FEWS export configuration for archiving database snap shots
Element | Format | Description |
general ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
ExportSnapShotActivity ComplexType |
|
|
areaId | string | area (folder) where snapshot is archived |
Figure 4.9 Delft-FEWS export configuration for archiving database snap shots