Versions Compared

Old Version 32

changes.mady.by.user Camiel van Breugel

Saved on

compared with

New Version 33

changes.mady.by.user Klaas-Jan van Heeringen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

With daily exports of observed data per area, the number of small datasets on the file system may increases quickly. This has several disadvantages as the harvesting process will take longer processing many small datasets, while the performance of seamless integration (and webservices) will also drop. Therefore, an amalgamate process is developed which merges the daily observation files into one observation file per month. Currently, this amalgamate process is executed from the Fews-application side by an FSS by using the following configuration. Specify the relativePeriod in such way that the observed datafiles are stable and are not updated anymore. E.g. use one or two months of delay

Code Block
languagexml
titleexample amalgamate observations
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd">
<amalgamateObserved>
    <general>
            <archiveFolder>$ARCHIVE_FOLDER$</archiveFolder>
        <relativePeriod unit="day" start="-62" end="-31"></relativePeriod>
    </general>
    <activities>
        <amalgamateObservedData>
            <areaId>$BASIN$</areaId>
        </amalgamateObservedData>
    </activities>
</amalgamateObserved>
</exportArchiveModule>


Element

Format

Description

GeneralAmalgamateObservedSection ComplexType

 

 

archiveFolder

string

Export destination folder, assumes that the account running the FEWS (FSS) application has write access

relativePeriod

 

period experiencing changed data that should be amalgamted

AmalgamateActivities ComplexType

 

 

amalgamateObservedData

AmalgamateActivity

indicator that observed data will be amalgamated

areaId

string

area for which amalgamation takes place

...


Delft-FEWS can export external forecast time series to the archive via the ArchiveExportModule (exportExternalForecast activity).
For external forecasts a dataset is generated for every area for every source for every forecast. The associated directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<yyyy>/<MM>/<areaId>/<dd> /external_forecasts/<sourceId>_<ExtForecastTime>
The exportArchiveModule.xsd has a dedicated exportExternalForecast section to configure the time series that need to be archived (see Figure 4.4). Table 4.3 documents the associated elements:

Figure 4.4 Delft-FEWS export configuration for archiving external forecasts


Table 4.3 Delft-FEWS export configuration for archiving external forecasts

...


Delft-FEWS can export the current configuration to the archive via the ArchiveExportModule (exportConfig activity). The configuration thus is exported as part of the workflow. The associated root directory structure of the Delft-FEWS export for this type of dataset is as follows:
<archiveRoot>/<config>/<areaId>/<yyyymmdd>/
The date refers to the revision date of the configuration. The file name typically holds the revisionId.
The exportArchiveModule.xsd has a dedicated exportConfig section to setup the export of the Configuration (see Figure 4.6). This is due to be revised as the same relativePeriod based export mechanism should be adopted for checking what to export (see Table 4.5).

Figure 4.6 Delft-FEWS export configuration for archiving configurations


Table 4.5 Delft-FEWS export configuration for archiving a configuration

...



Figure 4.7 Delft-FEWS export configuration for archiving rating curves, general section


Table 4.7and Figure 4.8 discuss the activities section of for exporting rating curves.

...


Figure 4.9 Delft-FEWS export configuration for archiving database snap shots


Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
    <exportSnapShot>
        <general>
            <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
        </general>
        <activities>
            <exportSnapShot>
                <areaId>area</areaId>
                <filter id="only time series">
                    <xmlConfig enabled="false" name="Default xml config" synchLevel="11"/>
                    <coldStates enabled="false" name="Default cold states" synchLevel="11"/>
                    <moduleDataSets enabled="false" name="Default module data sets" synchLevel="11"/>
                    <mapLayers enabled="false" name="Default map layers" synchLevel="11"/>
                    <icons enabled="false" name="Default icons" synchLevel="11"/>
                    <reportTemplates enabled="false" name="Default report templates" synchLevel="11"/>
                    <reportImages enabled="false" name="Default report images" synchLevel="11"/>
                    <continuousTimeSeries enabled="true" name="Simulated" synchLevel="0" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Telemetry" synchLevel="1" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Manual" synchLevel="5" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Astronomical and climatological" synchLevel="4" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Small external forecast grids" synchLevel="6" maxAge="1000" unit="week"/>
                    <continuousTimeSeries enabled="true" name="Large external forecast grids" synchLevel="16" maxAge="10000" unit="week"/>
                    <warmStates enabled="false" name="Warm states" maxAge="10" unit="week"/>
                    <logEntries enabled="false" name="Log Entries" maxAge="1" unit="week"/>
                    <thresholdEvents enabled="false" name="Threshold Events" maxAge="1" unit="week"/>
                </filter>
            </exportSnapShot>
        </activities>
    </exportSnapShot>
</exportArchiveModule>

Archiving Delft-FEWS products

...

An configuration example is given below.

Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>area</areaId>
            <sourceId>source</sourceId>
            <sourceFile>$INPUT_FOLDER$/sourceA</sourceFile>
            <sourceFile>$INPUT_FOLDER$/sourceB</sourceFile>
            <moveFile>true</moveFile>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...

An configuration example is given below.

Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>areaTest</areaId>
            <sourceId>sourceTest</sourceId>
            <sources>
               <zipProduct>true</zipProduct>
               <sourceFiles>$INPUT_FOLDER$/testdata/exportandmoveandzip/source?</sourceFiles>
               <relativePath>aa</relativePath>
            </sources>
            <moveFile>true</moveFile>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...

An configuration example is given below

Code Block
languagexml
<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule
      xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
   <exportProducts>
      <general>
         <archiveFolder>$ARCHIVE_DIR$</archiveFolder>
      </general>
      <activities>
         <exportProduct>
            <areaId>areaTest</areaId>
            <sourceId>sourceTest</sourceId>
            <importFolder>$IMPORT_DIR$</importFolder>
            <fileNameProductDateTimePattern>yyyyMMdd'.nc'</fileNameProductDateTimePattern>
         </exportProduct>
      </activities>
   </exportProducts>
</exportArchiveModule>

...