...
With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration. Specify the relativePeriod in such way that the observed datafiles are stable and are not updated anymore. E.g. use one or two months of delay
Code Block | ||||
---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="UTF-8"?> <exportArchiveModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"> <amalgamateObserved> <general> <archiveFolder>$ARCHIVE_FOLDER$</archiveFolder> <relativePeriod unit="day" start="-62" end="-31"></relativePeriod> </general> <activities> <amalgamateObservedData> <areaId>$BASIN$</areaId> </amalgamateObservedData> </activities> </amalgamateObserved> </exportArchiveModule> |
Element | Format | Description |
GeneralAmalgamateObservedSection ComplexType |
|
|
archiveFolder | string | Export destination folder, assumes that the account running the FEWS (FSS) application has write access |
relativePeriod |
| period experiencing changed data that should be amalgamted |
AmalgamateActivities ComplexType |
|
|
amalgamateObservedData | AmalgamateActivity | indicator that observed data will be amalgamated |
areaId | string | area for which amalgamation takes place |
...
Delft-FEWS can export external forecast time series to the archive via the ArchiveExportModule (exportExternalForecast activity).
For external forecasts a dataset is generated for every area for every source for every forecast. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd> /external_forecasts/<sourceId>_<ExtForecastTime>
The exportArchiveModule.xsd has a dedicated exportExternalForecast section to configure the time series that need to be archived (see Figure 4.4). Table 4.3 documents the associated elements:
Figure 4.4 Delft-FEWS export configuration for archiving external forecasts
Table 4.3 Delft-FEWS export configuration for archiving external forecasts
...
Delft-FEWS can export the current configuration to the archive via the ArchiveExportModule (exportConfig activity). The configuration thus is exported as part of the workflow. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<config>/<areaId>/<yyyymmdd>/
The date refers to the revision date of the configuration. The file name typically holds the revisionId.
The exportArchiveModule.xsd has a dedicated exportConfig section to setup the export of the Configuration (see Figure 4.6). This is due to be revised as the same relativePeriod based export mechanism should be adopted for checking what to export (see Table 4.5).
Figure 4.6 Delft-FEWS export configuration for archiving configurations
Table 4.5 Delft-FEWS export configuration for archiving a configuration
...
Figure 4.7 Delft-FEWS export configuration for archiving rating curves, general section
Table 4.7and Figure 4.8 discuss the activities section of for exporting rating curves.
...
Figure 4.9 Delft-FEWS export configuration for archiving database snap shots
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?> <exportArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews"> <exportSnapShot> <general> <archiveFolder>$ARCHIVE_DIR$</archiveFolder> </general> <activities> <exportSnapShot> <areaId>area</areaId> <filter id="only time series"> <xmlConfig enabled="false" name="Default xml config" synchLevel="11"/> <coldStates enabled="false" name="Default cold states" synchLevel="11"/> <moduleDataSets enabled="false" name="Default module data sets" synchLevel="11"/> <mapLayers enabled="false" name="Default map layers" synchLevel="11"/> <icons enabled="false" name="Default icons" synchLevel="11"/> <reportTemplates enabled="false" name="Default report templates" synchLevel="11"/> <reportImages enabled="false" name="Default report images" synchLevel="11"/> <continuousTimeSeries enabled="true" name="Simulated" synchLevel="0" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Telemetry" synchLevel="1" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Manual" synchLevel="5" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Astronomical and climatological" synchLevel="4" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Small external forecast grids" synchLevel="6" maxAge="1000" unit="week"/> <continuousTimeSeries enabled="true" name="Large external forecast grids" synchLevel="16" maxAge="10000" unit="week"/> <warmStates enabled="false" name="Warm states" maxAge="10" unit="week"/> <logEntries enabled="false" name="Log Entries" maxAge="1" unit="week"/> <thresholdEvents enabled="false" name="Threshold Events" maxAge="1" unit="week"/> </filter> </exportSnapShot> </activities> </exportSnapShot> </exportArchiveModule> |
Archiving Delft-FEWS products
...
An configuration example is given below.
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>area</areaId>
<sourceId>source</sourceId>
<sourceFile>$INPUT_FOLDER$/sourceA</sourceFile>
<sourceFile>$INPUT_FOLDER$/sourceB</sourceFile>
<moveFile>true</moveFile>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...
An configuration example is given below.
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>areaTest</areaId>
<sourceId>sourceTest</sourceId>
<sources>
<zipProduct>true</zipProduct>
<sourceFiles>$INPUT_FOLDER$/testdata/exportandmoveandzip/source?</sourceFiles>
<relativePath>aa</relativePath>
</sources>
<moveFile>true</moveFile>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...
An configuration example is given below
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
<exportProducts>
<general>
<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
</general>
<activities>
<exportProduct>
<areaId>areaTest</areaId>
<sourceId>sourceTest</sourceId>
<importFolder>$IMPORT_DIR$</importFolder>
<fileNameProductDateTimePattern>yyyyMMdd'.nc'</fileNameProductDateTimePattern>
</exportProduct>
</activities>
</exportProducts>
</exportArchiveModule> |
...