| scrollbar |
|---|
| Anchor | ||||
|---|---|---|---|---|
|
...
For observations a dataset is generated for every area on a daily basis. The associated directory structure of the Delft-FEWS export for this type of data set is as follows:
<archive root>/<yyyy>/<MM>/<areaId>/<dd>/observed/
When Delft-FEWS generates the netcdf file, data is written to the same data block when the entire matrix is filled, i.e. all time steps are regular and none of the values is missing. For those locations with irregular time stamps, or missing values, a separate data block is used.
Within the netCDF file, each data block is accompanied by a header. Within each header a metadata item called 'timeseries_sets_xml' is included, holding the exact definition of the timeSeriesSet as the data was stored in the FEWS database. This feature allows full reproducibility of the time series via an Import workflow in a Delft-FEWS stand alone application, assuming that the associated Delft-FEWS configuration is in place.
The relative period used in the export of observed data sets requires some additional explanation. The relative period defines which daily data sets are written to archive. The export of observed data is usually scheduled once a day after midnight when all the data for the prior day are imported and available in FEWS. Because it is common to configure a data export of several days this means that data sets which are written earlier will be overwritten. This is done for several reasons. First all this ensure that when a data export was not executed for several days because of an system error this missing data sets will be added during the next run. In addition this will also allow that data edits will be stored in the archive. To ensure that data edits will be stored in the archive you must be sure that time window which you configure in your relative period is large enough to capture the data edits. However it very important that the relative period will be is not too large! If the beginning of the relative period starts at a time that data is already expired from the archive you will write missing values to the archive and erase the data in your archive! A relative period of -10 to -1 is usually sufficient for most systems.
...
Element | Format | Description | ||
general ComplexType |
|
| ||
| dataSetPeriod | string | By default the daily data sets are created. It also possible to create monthly data sets. For operational purposes it is recommend to use daily data sets. For migrating data from other systems to the FEWS archive it might be convenient to export the data in montly data sets. | ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod |
| Exports entire the dataset by day. | ||
idMap | string | idMap applied to translate internal FEWS identifiers to identifiers that meet NetCDF-CF criteria.E.g. netcdf does not allow a full stop ('.') in the variable name | ||
| ignoreNonExistingLocationSets | bool | If this option is set FEWS will not log an error when an location set is not configured. | ||
| verifyExportedTimeSeries | bool | if this option is set FEWS will verify if data in the exported netcdf is the same as in the FEWS database. | ||
netcdfObservedExportActivities ComplexType |
|
| ||
fileName | string | should include nc extension, otherwise files will not be read. preferably no spaces | ||
areaId | string | area to which the dataset belongs | ||
| sourceId | string | defines the source of the data set | ||
| exportLocationAttributeAsNetCDFVariable | complex type | Since 2021.01. Adds a variable to the NetCdf file. Name of the variable is the value of ncVariable, the value is the configured location attribute of attributeId. | ||
ncMetaData | string elem. | optional metadata tags within NetCDF file following CF convention. Supported by the internal catalogue of the THREDDS Data Server | ||
includeFlags | bool | default=TRUE; if TRUE, a list of flags is stored, each value pointing to the associated flag | ||
| includeTimeSeriesProperties | bool | default=TRUE, if TRUE the properties of the time series will also be stored in the NetCDF file. | ||
includeComments | bool | default=TRUE; if TRUE, a list of comments is stored, each value pointing to the associated comment | ||
thresholdGroupId | string | identifies FEWS ThresholdGroup which is used to detect threshold crossings to be highlighted in the metaData.xml | ||
timeSeriesSets | FEWS timeseries sets |
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file. 
Figure 4.2 Deft-FEWS export configuration for archiving observations
...
Element | Format | Description | ||
GeneralAmalgamateObservedSection ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod |
| period experiencing changed data that should be amalgamted | ||
AmalgamateActivities ComplexType |
|
| ||
amalgamateObservedData | AmalgamateActivity | indicator that observed data will be amalgamated | ||
areaId | string | area for which amalgamation takes place |
...
Element | Format | Description | ||
general ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod |
| Exports entire dataset by day, for any day where a database change (blob creation time) is detected within the relativePeriod (relative to T0). Existing files are overwritten* | ||
MessagesActivities ComplexType |
|
| ||
forecasterNotesExportActivity ComplexType* |
|
| ||
areaId | string | area for which messages need to be archived |
...
Element | Format | Description | ||
GeneralExportForecastSection ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod |
| Exports entire external forecast dataset by source and forecast time, for any forecast where a database change (blob creation time) is detected within the relativePeriod (relative to T0). Existing timeseries files are overwritten* | ||
idMap | string | idMap applied to translate internal FEWS identifiers to identifiers that meet NetCDF-CF criteria.E.g. netcdf does not allow a full stop ('.') in the variable name | ||
ExternalForecastActivities ComplexType |
|
| ||
NetcdfExternalForecast ExportActivities ComplexType |
|
| ||
NetcdfExternalForecast ExportActivity ComplexType* |
| |||
fileName | string | without nc extension, preferably no spaces | ||
areaId | string | area to which the dataset belongs | ||
sourceId |
| identifies underlying source e.g. NWP product | ||
ncMetaData | string elem. | optional metadata tags within NetCDF file following CF convention. Supported by the internal catalogue of the THREDDS Data Server | ||
includeFlags | bool | only applied for scalar values. | ||
includeComments | bool | only applied for scalar values. | ||
| includeTimeSeriesProperties | bool | only applied for scalar values. default=TRUE; if TRUE, a list of properties is stored, with each value pointing to the associated property | ||
| includeTimeRanges | bool | only applied for nonequidistant scalar values. Not valid for ensembles. default = TRUE; if TRUE, time-start and time-end properties will be stored if available. | ||
| includeValueRanges | bool | only applied for nonequidistant scalar values. Not valid for ensembles. default = TRUE; if TRUE, value-start and value-end properties will be stored if available. | ||
timeSeriesSets |
| FEWS timeseries sets |
*When an existing file is locked while it needs to be overwritten, the export function writes a new temporary file. The FileSweeper, a scheduled process, renames this file when the lock is removed from the original file.
...
Element | Format | Description | ||
GeneralExportForecastSection ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod |
| Exports entire simulated timeseries by workflow. | ||
| workflowId | string | This option should be in combination with the relativePeriod. Al the forecasts of the workflowId which exists in the relative period will be exported | ||
idMap | string | idMap applied to translate internal FEWS identifiers to identifiers that meet NetCDF-CF criteria.E.g. netcdf does not allow a full stop ('.') in the variable name | ||
| relatedWorkflowId | String | If the export is done without the relativePeriod and workflowId option but not in the workflow which created the forecasts the workflow id of the forecast you are trying to export should be configured to make sure that the modifiers which are used in this forecast will be exported. | ||
ForecastActivities ComplexType |
|
| ||
NetcdfForecastExportActivities ComplexType |
|
| ||
NetcdfForecastExportActivity ComplexType* |
|
| ||
fileName | string | without nc extension, preferably no spaces | ||
areaId | string | area to which the dataset belongs | ||
ncMetaData | string elem. | optional metadata tags within NetCDF file following CF convention. Supported by the internal catalogue of the THREDDS Data Server | ||
includeFlags | bool | only applied for scalar values. | ||
includeComments | bool | only applied for scalar values. | ||
timeSeriesSets | FEWS timeseries sets | |||
ReportsExportActivity ComplexType* |
| |||
subfolder | string | Subdirectory within the 'reports' directory | ||
moduleInstanceId | string | moduleInstanceId which created the report | ||
ModuleStatesExportActivity ComplexType* |
|
| ||
moduleInstanceId | string | moduleInstanceId which created the state |
...
Element | Format | Description | ||
GeneralExportConfig ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod (TO DO) | Exports entire configuration when database change (config revision) has been detected within the relativePeriod (relative to T0). Existing files are overwritten* | |||
ExportConfigActivities ComplexType |
| |||
ExportConfigActivity ComplexType* |
|
| ||
areaId | string | area to which the dataset belongs |
...
Element | Format | Description | ||
GeneralExportRatingCurves ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
relativePeriod |
| If defined, only rating curves which have database changes during the relativePeriod (relative to T0) are exported. If no relativePeriod is specified all available rating curves are exported | ||
Prefix-timeZeroFormattingString | string | Adds T0 stamp as prefix to file name E.g.yyyymmdd_hhmm | ||
idMap | string | idMap applied to translate internal FEWS identifiers to external identifiers | ||
unitConversionsId | string | Allows conversion of rating table units | ||
includeModifiers | bool | Default=FALSE; If TRUE, rating curve modifiers should be included in the export |
...
Element | Format | Description | ||
RatingCurvesActivities ComplexType |
| |||
RatingCurvesExportActivity ComplexType |
|
| ||
fileName | string | without extension, preferably no spaces | ||
areaId | string | area to which the dataset belongs | ||
linearTableStageresolution | string | Stage increment between the rows in a rating table | ||
locationId/locationSetId | string | Locations for which rating curves should be exported to this file |
...
Element | Format | Description | ||
general ComplexType |
|
| ||
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access | ||
| zipExportedSnapShot | boolean | Place local datastore in zip file. Default true | ||
| singularExport | The archive folder structure will be ignored and the snapshot will be written directly in the archiveFolder | |||
ExportSnapShotActivity ComplexType |
|
| ||
areaId | string | area (folder) where snapshot is archived |
...
It is possible to define individual files which should be archived as part of the product. But it is also possible to archive files based on a wild card.
Table 4.8 Delft-FEWS export configuration for archiving database snap shots
Element | Format | Description | ||
general ComplexType |
|
| ||
archiveFolder | ||||
ExportProductActivityComplexType |
| |||
areaId | string | Id of the area to which this product belongs | ||
| sourceId | string | Id of the source to which this product belongs | ||
| sourceFile | string | File which should be archived as part of the product | ||
| moveFile | boolean | If this option is set to true the source file will be deleted after it has been archived. | ||
| sources/zipProduct | boolean | if this option is set to true the files will | ||
| sources/sourceFiles | string | |||
| sources/relativePath | string | the files identified by the tag sourceFiles will be exported to folder with the configurewithin the main folder of the data set identfied | ||
| moveFile | boolean | If this option is set to true then the source file will be deleted after the file has been copied to the archive |
An configuration example is given below.
| Code Block | ||
|---|---|---|
| ||
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>area</areaId>
<sourceId>source</sourceId>
<sourceFile>$INPUT_FOLDER$/sourceA</sourceFile>
<sourceFile>$INPUT_FOLDER$/sourceB</sourceFile>
<moveFile>true</moveFile>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...
An configuration example is given below.
| Code Block | ||
|---|---|---|
| ||
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>areaTest</areaId>
<sourceId>sourceTest</sourceId>
<sources>
<zipProduct>true</zipProduct>
<sourceFiles>$INPUT_FOLDER$/testdata/exportandmoveandzip/source?</sourceFiles>
<relativePath>aa</relativePath>
</sources>
<moveFile>true</moveFile>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...
Archiving products by defining an import folder and a date/time pattern
...
| Code Block | ||
|---|---|---|
| ||
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>areaTest</areaId>
<sourceId>sourceTest</sourceId>
<importFolder>$IMPORT_DIR$</importFolder>
<fileNameProductDateTimePattern>yyyyMMdd'.nc'</fileNameProductDateTimePattern>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...