Page History

...

...

With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration. Specify the relativePeriod in such way that the observed datafiles are stable and are not updated anymore. E.g. use oen one or two montsh months of delay

<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd">
<amalgamateObserved>
    <general>
            <archiveFolder>$ARCHIVE_FOLDER$</archiveFolder>
        <relativePeriod unit="day" start="-62" end="-31"></relativePeriod>
    </general>
    <activities>
        <amalgamateObservedData>
            <areaId>$BASIN$</areaId>
        </amalgamateObservedData>
    </activities>
</amalgamateObserved>
</exportArchiveModule>

...

Delft-FEWS can export messages to the archive via the ArchiveExportModule (exportMessages activity).
For messages a dataset is generated for every area on a daily basis. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows.
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd>/messages/ The exportArchiveModule.xsd has a dedicated exportMessages section to configure the messages that need to be archived (see Figure 4.3). Currently only ForecasterNotes can be archived. Table 4.2 documents the associated elements:

Note: The date associated with a message is the creation time, not the T0 at which the forecaster note was created or the Event Time. This means that the relative period as configured in exportMessages  (which is relative to T0), for exporting the forecaster Notes to the Archive should cover the creation time. This is not so mch an issue in a live system, as it is in a Stand Alone, where T0 and System Time might be far apart.

The exportArchiveModule.xsd has a dedicated exportMessages section to configure the messages that need to be archived (see Figure 4.3). Currently only ForecasterNotes can be archived. Table 4.2 documents the associated elements:
Table 4.Table 4.2 Delft-FEWS export configuration for archiving messages

...

Delft-FEWS can export external forecast time series to the archive via the ArchiveExportModule (exportExternalForecast activity).
For external forecasts a dataset is generated for every area for every source for every forecast. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd> /external_forecasts/<sourceId>_<ExtForecastTime>
The exportArchiveModule.xsd has a dedicated exportExternalForecast section to configure the time series that need to be archived (see Figure 4.4). Table 4.3 documents the associated elements:

Figure 4.4 Delft-FEWS export configuration for archiving external forecasts

Table 4.3 Delft-FEWS export configuration for archiving external forecasts

...

Delft-FEWS can export the current configuration to the archive via the ArchiveExportModule (exportConfig activity). The configuration thus is exported as part of the workflow. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<config>/<areaId>/<yyyymmdd>/
The date refers to the revision date of the configuration. The file name typically holds the revisionId.
The exportArchiveModule.xsd has a dedicated exportConfig section to setup the export of the Configuration (see Figure 4.6). This is due to be revised as the same relativePeriod based export mechanism should be adopted for checking what to export (see Table 4.5).

Figure 4.6 Delft-FEWS export configuration for archiving configurations

Table 4.5 Delft-FEWS export configuration for archiving a configuration

...

Delft-FEWS can export rating curves to the archive via the ArchiveExportModule (exportRatingCurves activity). The entire history of the rating curves is exported . The and the export format is PI-XML (not NetCDF!). The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows FEWS-10198: update directory structure to above setup:
<archiveRoot>/ratingcurves/<areaId>
The configurator can choose between exporting the full set or just the rating curves that have changed, with or without modifier changes.
The exportArchiveModule.xsd has a dedicated exportRatingCurve section to setup the export of the Configuration. Table 4.6 and Figure 4.7 discuss the general section.

...

Figure 4.7 Delft-FEWS export configuration for archiving rating curves, general section

Table 4.7and Figure 4.8 discuss the activities section of for exporting rating curves.

...

...

Anchor
_Toc386122339 _Toc386122339

Archiving Delft-FEWS snap shots

Location of the snapshot

Snap shots of the Delft-FEWS database can be archived via the archiveExportModule (exportSnapShot activity).
For snapshots a dataset is generated and stored under the configured area at the memoment moment of workflow execution. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId/<snapshot>/
Figure 4.9 and Table 4.8 documents the associated elements in the exportSnapShot activity:.

Incremental export

Since 2019.02 the archive snapshot export is incremental. A copy of the last exported snapshot is made to use as base for the new export. The modified and new rows are copied. Rows that are no longer in the specified profile or central database are deleted from the copy. The snapshot is compacted afterwards . Whether or not a previous snapshot is used as base, the created snapshot contains the same rows and has the same size.

Delft-FEWS stores the location of the last snapshot in a text file: $ARCHIVE_FOLDER$\lastExportedSnapshotPath_$AREA$.txt, If this file does not exist, Delft-FEWS will export a complete snapshot.

Table 4.8 Delft-FEWS export configuration for archiving database snap shots

Figure 4.9 Delft-FEWS export configuration for archiving database snap shotsexport configuration for archiving database snap shots

<exportArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
    <exportSnapShot>
        <general>
            <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
        </general>
        <activities>
            <exportSnapShot>
                <areaId>area</areaId>
                <filter id="only time series">
                    <xmlConfig enabled="false" name="Default xml config" synchLevel="11"/>
                    <coldStates enabled="false" name="Default cold states" synchLevel="11"/>
                    <moduleDataSets enabled="false" name="Default module data sets" synchLevel="11"/>
                    <mapLayers enabled="false" name="Default map layers" synchLevel="11"/>
                    <icons enabled="false" name="Default icons" synchLevel="11"/>
                    <reportTemplates enabled="false" name="Default report templates" synchLevel="11"/>
                    <reportImages enabled="false" name="Default report images" synchLevel="11"/>
                    <continuousTimeSeries enabled="true" name="Simulated" synchLevel="0" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Telemetry" synchLevel="1" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Manual" synchLevel="5" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Astronomical and climatological" synchLevel="4" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Small external forecast grids" synchLevel="6" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Large external forecast grids" synchLevel="16" maxAge="10000" unit="week"/>
                    <warmStates enabled="false" name="Warm states" maxAge="10" unit="week"/>
                    <logEntries enabled="false" name="Log Entries" maxAge="1" unit="week"/>
                    <thresholdEvents enabled="false" name="Threshold Events" maxAge="1" unit="week"/>
                </filter>
            </exportSnapShot>
        </activities>
    </exportSnapShot>
</exportArchiveModule>

Archiving Delft-FEWS products

...

An configuration example is given below.

<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>area</areaId>
            <sourceId>source</sourceId>
            <sourceFile>$INPUT_FOLDER$/sourceA</sourceFile>
            <sourceFile>$INPUT_FOLDER$/sourceB</sourceFile>
            <moveFile>true</moveFile>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...

An configuration example is given below.

<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>areaTest</areaId>
            <sourceId>sourceTest</sourceId>
            <sources>
               <zipProduct>true</zipProduct>
               <sourceFiles>$INPUT_FOLDER$/testdata/exportandmoveandzip/source?</sourceFiles>
               <relativePath>aa</relativePath>
            </sources>
            <moveFile>true</moveFile>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...

An configuration example is given below

<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>areaTest</areaId>
            <sourceId>sourceTest</sourceId>
            <importFolder>$IMPORT_DIR$</importFolder>
            <fileNameProductDateTimePattern>yyyyMMdd'.nc'</fileNameProductDateTimePattern>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...