Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration. Specify the relativePeriod in such way that the observed datafiles are stable and are not updated anymore. E.g. use one or two months of delay

Code Block
languagexml
titleexample amalgamate observations
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd">
<amalgamateObserved>
    <general>
            <archiveFolder>$ARCHIVE_FOLDER$</archiveFolder>
        <relativePeriod unit="day" start="-62" end="-31"></relativePeriod>
    </general>
    <activities>
        <amalgamateObservedData>
            <areaId>$BASIN$</areaId>
        </amalgamateObservedData>
    </activities>
</amalgamateObserved>
</exportArchiveModule>



Element

Format

Description

GeneralAmalgamateObservedSection ComplexType

 

 

archiveFolder

string

Export destination folder, assumes that the account running the FEWS (FSS) application has write access

relativePeriod

 

period experiencing changed data that should be amalgamted

AmalgamateActivities ComplexType

 

 

amalgamateObservedData

AmalgamateActivity

indicator that observed data will be amalgamated

areaId

string

area for which amalgamation takes place

...


Delft-FEWS can export external forecast time series to the archive via the ArchiveExportModule (exportExternalForecast activity).
For external forecasts a dataset is generated for every area for every source for every forecast. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd> /external_forecasts/<sourceId>_<ExtForecastTime>
The exportArchiveModule.xsd has a dedicated exportExternalForecast section to configure the time series that need to be archived (see Figure 4.4). Table 4.3 documents the associated elements:

Figure 4.4 Delft-FEWS export configuration for archiving external forecasts


Table 4.3 Delft-FEWS export configuration for archiving external forecasts

...


Delft-FEWS can export the current configuration to the archive via the ArchiveExportModule (exportConfig activity). The configuration thus is exported as part of the workflow. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<config>/<areaId>/<yyyymmdd>/
The date refers to the revision date of the configuration. The file name typically holds the revisionId.
The exportArchiveModule.xsd has a dedicated exportConfig section to setup the export of the Configuration (see Figure 4.6). This is due to be revised as the same relativePeriod based export mechanism should be adopted for checking what to export (see Table 4.5).

Figure 4.6 Delft-FEWS export configuration for archiving configurations


Table 4.5 Delft-FEWS export configuration for archiving a configuration

...


Delft-FEWS can export rating curves to the archive via the ArchiveExportModule (exportRatingCurves activity). The entire history of the rating curves is exported and the export format is PI-XML (not NetCDF!). The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows FEWS-10198: update directory structure to above setup:
<archiveRoot>/ratingcurves/<areaId>
The configurator can choose between exporting the full set or just the rating curves that have changed, with or without modifier changes.
The exportArchiveModule.xsd has a dedicated exportRatingCurve section to setup the export of the Configuration. Table 4.6 and Figure 4.7 discuss the general section.

...



Figure 4.7 Delft-FEWS export configuration for archiving rating curves, general section


Table 4.7and Figure 4.8 discuss the activities section of for exporting rating curves.

...

Element

Format

Description

RatingCurvesActivities ComplexType

 

 

RatingCurvesExportActivity ComplexType

 

 

fileName

string

without nc extension, preferably no spaces

areaId

string

area to which the dataset belongs

linearTableStageresolution

string

Stage increment between the rows in a rating table

locationId/locationSetId

string

Locations for which rating curves should be exported to this file

...

Anchor
_Toc386122339
_Toc386122339
Archiving Delft-FEWS snap shots

Location of the snapshot

Snap shots of the Delft-FEWS database can be archived via the archiveExportModule (exportSnapShot activity).
For snapshots a dataset is generated and stored under the configured area at the memoment moment of workflow execution. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId/<snapshot>/
Figure 4.9 and Table 4.8 documents the associated elements in the exportSnapShot activity:.

Incremental export

Since 2019.02 the archive snapshot export is incremental. A copy of the last exported snapshot is made to use as base for the new export. The modified and new rows are copied. Rows that are no longer in the specified profile or central database are deleted from the copy. The snapshot is compacted afterwards . Whether or not a previous snapshot is used as base, the created snapshot contains the same rows and has the same size.

Delft-FEWS stores the location of the last snapshot in a text file: $ARCHIVE_FOLDER$\lastExportedSnapshotPath_$AREA$.txt, If this file does not exist, Delft-FEWS will export a complete snapshot.


Table 4.8 Delft-FEWS export configuration for archiving database snap shots

Element

Format

Description

general ComplexType

 

 

archiveFolder

string

Export destination folder, assumes that the account running the FEWS (FSS) application has write access

zipExportedSnapShotbooleanPlace local datastore in zip file. Default true
singularExport
The archive folder structure will be ignored and the snapshot will be written directly in the archiveFolder

ExportSnapShotActivity ComplexType

 

 

areaId

string

area (folder) where snapshot is archived

...


Figure 4.9 Delft-FEWS export configuration for archiving database snap shots


Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
    <exportSnapShot>
        <general>
            <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
        </general>
        <activities>
            <exportSnapShot>
                <areaId>area</areaId>
                <filter id="only time series">
                    <xmlConfig enabled="false" name="Default xml config" synchLevel="11"/>
                    <coldStates enabled="false" name="Default cold states" synchLevel="11"/>
                    <moduleDataSets enabled="false" name="Default module data sets" synchLevel="11"/>
                    <mapLayers enabled="false" name="Default map layers" synchLevel="11"/>
                    <icons enabled="false" name="Default icons" synchLevel="11"/>
                    <reportTemplates enabled="false" name="Default report templates" synchLevel="11"/>
                    <reportImages enabled="false" name="Default report images" synchLevel="11"/>
                    <continuousTimeSeries enabled="true" name="Simulated" synchLevel="0" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Telemetry" synchLevel="1" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Manual" synchLevel="5" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Astronomical and climatological" synchLevel="4" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Small external forecast grids" synchLevel="6" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Large external forecast grids" synchLevel="16" maxAge="10000" unit="week"/>
                    <warmStates enabled="false" name="Warm states" maxAge="10" unit="week"/>
                    <logEntries enabled="false" name="Log Entries" maxAge="1" unit="week"/>
                    <thresholdEvents enabled="false" name="Threshold Events" maxAge="1" unit="week"/>
                </filter>
            </exportSnapShot>
        </activities>
    </exportSnapShot>
</exportArchiveModule>


Archiving Delft-FEWS products

...

An configuration example is given below.

Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>area</areaId>
            <sourceId>source</sourceId>
            <sourceFile>$INPUT_FOLDER$/sourceA</sourceFile>
            <sourceFile>$INPUT_FOLDER$/sourceB</sourceFile>
            <moveFile>true</moveFile>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...

An configuration example is given below.

<?xml version="1.0" encoding="UTF-8"?>
Code Block
languagexml
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>areaTest</areaId>
            <sourceId>sourceTest</sourceId>
            <sources>
               <zipProduct>true</zipProduct>
               <sourceFiles>$INPUT_FOLDER$/testdata/exportandmoveandzip/source?</sourceFiles>
               <relativePath>aa</relativePath>
            </sources>
            <moveFile>true</moveFile>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...

An configuration example is given below

Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>areaTest</areaId>
            <sourceId>sourceTest</sourceId>
            <importFolder>$IMPORT_DIR$</importFolder>
            <fileNameProductDateTimePattern>yyyyMMdd'.nc'</fileNameProductDateTimePattern>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...