...
With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration. Specify the relativePeriod in such way that the observed datafiles are stable and are not updated anymore. E.g. use one or two months of delay
| Code Block | ||||
|---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="UTF-8"?> <exportArchiveModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"> <amalgamateObserved> <general> <archiveFolder>$ARCHIVE_FOLDER$</archiveFolder> <relativePeriod unit="day" start="-62" end="-31"></relativePeriod> </general> <activities> <amalgamateObservedData> <areaId>$BASIN$</areaId> </amalgamateObservedData> </activities> </amalgamateObserved> </exportArchiveModule> |
Element | Format | Description |
GeneralAmalgamateObservedSection ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| period experiencing changed data that should be amalgamted |
AmalgamateActivities ComplexType |
|
|
amalgamateObservedData | AmalgamateActivity | indicator that observed data will be amalgamated |
areaId | string | area for which amalgamation takes place |
...
Delft-FEWS can export external forecast time series to the archive via the ArchiveExportModule (exportExternalForecast activity).
For external forecasts a dataset is generated for every area for every source for every forecast. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd> /external_forecasts/<sourceId>_<ExtForecastTime>
The exportArchiveModule.xsd has a dedicated exportExternalForecast section to configure the time series that need to be archived (see Figure 4.4). Table 4.3 documents the associated elements:
Figure 4.4 Delft-FEWS export configuration for archiving external forecasts
Table 4.3 Delft-FEWS export configuration for archiving external forecasts
...
Delft-FEWS can export the current configuration to the archive via the ArchiveExportModule (exportConfig activity). The configuration thus is exported as part of the workflow. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<config>/<areaId>/<yyyymmdd>/
The date refers to the revision date of the configuration. The file name typically holds the revisionId.
The exportArchiveModule.xsd has a dedicated exportConfig section to setup the export of the Configuration (see Figure 4.6). This is due to be revised as the same relativePeriod based export mechanism should be adopted for checking what to export (see Table 4.5).
Figure 4.6 Delft-FEWS export configuration for archiving configurations
Table 4.5 Delft-FEWS export configuration for archiving a configuration
...
Delft-FEWS can export rating curves to the archive via the ArchiveExportModule (exportRatingCurves activity). The entire history of the rating curves is exported and the export format is PI-XML (not NetCDF!). The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows FEWS-10198: update directory structure to above setup:
<archiveRoot>/ratingcurves/<areaId>
The configurator can choose between exporting the full set or just the rating curves that have changed, with or without modifier changes.
The exportArchiveModule.xsd has a dedicated exportRatingCurve section to setup the export of the Configuration. Table 4.6 and Figure 4.7 discuss the general section.
...
Figure 4.7 Delft-FEWS export configuration for archiving rating curves, general section
Table 4.7and Figure 4.8 discuss the activities section of for exporting rating curves.
...
Element | Format | Description |
RatingCurvesActivities ComplexType |
|
|
RatingCurvesExportActivity ComplexType |
|
|
fileName | string | without nc extension, preferably no spaces |
areaId | string | area to which the dataset belongs |
linearTableStageresolution | string | Stage increment between the rows in a rating table |
locationId/locationSetId | string | Locations for which rating curves should be exported to this file |
...
| Anchor | ||||
|---|---|---|---|---|
|
Location of the snapshot
Snap shots of the Delft-FEWS database can be archived via the archiveExportModule (exportSnapShot activity).
For snapshots a dataset is generated and stored under the configured area at the memoment moment of workflow execution. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId/<snapshot>/
Figure 4.9 and Table 4.8 documents the associated elements in the exportSnapShot activity:.
Incremental export
Since 2019.02 the archive snapshot export is incremental. A copy of the last exported snapshot is made to use as base for the new export. The modified and new rows are copied. Rows that are no longer in the specified profile or central database are deleted from the copy. The snapshot is compacted afterwards . Whether or not a previous snapshot is used as base, the created snapshot contains the same rows and has the same size.
Table 4.8 Delft-FEWS export configuration for archiving database snap shots
Delft-FEWS stores the location of the last snapshot in a text file: $ARCHIVE_FOLDER$\lastExportedSnapshotPath_$AREA$.txt, If this file does not exist, Delft-FEWS will export a complete snapshot.
Table 4.8 Delft-FEWS export configuration for archiving database snap shots
Element | Format | Description |
Element | Format | Description |
general ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
| zipExportedSnapShot | boolean | Place local datastore in zip file. Default true |
| singularExport | The archive folder structure will be ignored and the snapshot will be written directly in the archiveFolder | |
ExportSnapShotActivity ComplexType |
|
|
areaId | string | area (folder) where snapshot is archived |
...
Figure 4.9 Delft-FEWS export configuration for archiving database snap shots
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?> <exportArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews"> <exportSnapShot> <general> <archiveFolder>$ARCHIVE_DIR$</archiveFolder> </general> <activities> <exportSnapShot> <areaId>area</areaId> <filter id="only time series"> <xmlConfig enabled="false" name="Default xml config" synchLevel="11"/> <coldStates enabled="false" name="Default cold states" synchLevel="11"/> <moduleDataSets enabled="false" name="Default module data sets" synchLevel="11"/> <mapLayers enabled="false" name="Default map layers" synchLevel="11"/> <icons enabled="false" name="Default icons" synchLevel="11"/> <reportTemplates enabled="false" name="Default report templates" synchLevel="11"/> <reportImages enabled="false" name="Default report images" synchLevel="11"/> <continuousTimeSeries enabled="true" name="Simulated" synchLevel="0" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Telemetry" synchLevel="1" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Manual" synchLevel="5" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Astronomical and climatological" synchLevel="4" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Small external forecast grids" synchLevel="6" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Large external forecast grids" synchLevel="16" maxAge="10000" unit="week"/> <warmStates enabled="false" name="Warm states" maxAge="10" unit="week"/> <logEntries enabled="false" name="Log Entries" maxAge="1" unit="week"/> <thresholdEvents enabled="false" name="Threshold Events" maxAge="1" unit="week"/> </filter> </exportSnapShot> </activities> </exportSnapShot> </exportArchiveModule> |
Archiving Delft-FEWS products
...
An configuration example is given below.
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>area</areaId>
<sourceId>source</sourceId>
<sourceFile>$INPUT_FOLDER$/sourceA</sourceFile>
<sourceFile>$INPUT_FOLDER$/sourceB</sourceFile>
<moveFile>true</moveFile>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...
An configuration example is given below.
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>areaTest</areaId>
<sourceId>sourceTest</sourceId>
<sources>
<zipProduct>true</zipProduct>
<sourceFiles>$INPUT_FOLDER$/testdata/exportandmoveandzip/source?</sourceFiles>
<relativePath>aa</relativePath>
</sources>
<moveFile>true</moveFile>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...
An configuration example is given below
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>areaTest</areaId>
<sourceId>sourceTest</sourceId>
<importFolder>$IMPORT_DIR$</importFolder>
<fileNameProductDateTimePattern>yyyyMMdd'.nc'</fileNameProductDateTimePattern>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...