Introduction

Currently, the only graphical user interface to access the Deltares Open Archive is via the Delft-FEWS GUI. In addition, data can be approached via the Geonetwork web client of via THREDDS web server.

...

This user guide is derived from the HyFS application as used at the Bureau of Meteorology.

Archived data

In the Delft-FEWS application, scheduled and manual archive workflows can take care of archiving the data (sets) from the database. The scheduled archive workflow is generally run once a day and will generally archive the following type of data:

...

Note: External data which is archived elsewhere should in general not be archived within the context of the Delft-FEWS application.

Events

Within Forecasting organisations, events are situations that can be distinguished from business as ususal as something special or interesting is happening, e.g. a flood, drought or calamity. Generally the operations during an event is hectic and no one wants to spent time on data archiving. however, after the event the archived data may be crucial for legal inquiries, performance assessment, or model calibration or to use as input for training. Therefore, the concept of Event has been introduced to label a period of interest for a certain area, such that one easily can find this data back.

Many of the important data sets stored in a Deltares Archive associated with a real time system can be organized in events, this is a special type of metadata overarching one or multiple data sets. The event type, area and period covered by the event determine which data sets belong to an event. Each event type will include one or multiple data sets. Typical event types within a real time application are:

• Historic Event: This event type contains observed time series that are stored in the archive. When a new historic event is made the data include in the historic event period will become available as historic event in the Delft-FEWS operator client and can be viewed together with the current observed or simulated data.
• Review Event: This event type contains may contain observed time series, external forecast time series, simulated time series, modifiers and messages. When a new review event type is made the data sets covered will be tagged and can be downloaded to a Delft-FEWS Stand Alone for post event review purposes.
• Flood Watch Event: This event type contains observed time series and external forecast time series. When a new flood watch event type is made the data sets covered will be tagged and can be downloaded to a Delft-FEWS Stand Alone for skill perfomance analysis
• Watercoach/Training Event: This event type contains observed time series, external forecast time series, simulated time series, modifiers, model states and messages. When a new water coach event type is made the data sets covered will be tagged and can be downloaded to a Delft-FEWS Stand Alone. The data downloaded can then be used for making a new Delft-FEWS Water Coach script for training purposes.

Event types can be defined as appropriate, but need to be registered on the archiver server side.

Additional to tagging and an event in the Deltares archive it is also possible to have attachments uploaded and linked to an event in the archive. Creating an event, downloading events and data sets, and attaching attachments to events can be done with the Delft-FEWS Archive display.

Finally, events can play a crucial role in the data management of the archive. Data labelled for a specific type of event may be help longer in the archive compared to data not labelled for an event. E.g. Numerical Weather Predictions may be removed quickly, except for those external forecast that are part of a Flood Watch or training event.

Archive display

All data that is archived can be accessed and downloaded through the Delft-FEWS archive display. The archive display can be started from the Delft-FEWS Explorer from the Tools menu.

Figure Delft-FEWS Archive display; create new event

When the display is initialised initialized it will try to contact the archive file server to download information about the archive content (the archive configuration).

Note: When the Delft-FEWS archive display cannot connect to the archive file server, there will be a error message shown when starting the display. If this is the case, make sure the computer has a network connection, access to the archive file server, and that the path to the archive file server is correctly configured in the global.properties file.

...

In the following paragraphs the functionality of the different tabs is explained.

Create an archive event

Events are created for Areas and are valid for a period; specified by the start and end time.

Figure Delft-FEWS Archive display; create new event

To create a new event; take the following steps:

...

4         Press the search button on the right part of the display to check if there is any data in the archive for the selected area and period.

5         Press the save event button to save the event file on the local computer. The event will be added as a new line in the event list table in red.

6         Press the upload event button to upload the event file from the local computer to the archive server.

Image Added

Figure Delft-FEWS Archive display; create new event 

When an event is created it is possible to attach a file (pdf file, or zip file, max. 100Mb) to the event. Take the following steps to attach an attachment to an existing event.

...

3         Press the upload event button to upload the event file from the local computer to the archive server.

Note: By double clicking one of the cells in the event list, the event list will be filtered and show only those events that have the same content of the selected cell. To filter for example on historic events, double click in the event type cell where the content is historic event. All cells in that event table column will now have a blue background color indicating there is a filter active. To remove the filter, double click again on one of the event type cells.

Search and Download archive events

The Deltares archive will contain many events for all catchments. The events can be searched and downloaded in the Search for Events tab.

Figure Delft-FEWS Archive display; search for events

To search for events; take the following steps:

...

5         Press the download event with data button to download the data for the selected events from the archive server to the local computer.

Note: By double clicking one of the cells in the event list, the event list will be filtered and show only those events that have the same content of the selected cell. To filter for example on historic events, double click in the event type cell where the content is historic event. All cells in that event table column will now have a blue background color indicating there is a filter active. To remove the filter, double click again on one of the event type cells.

Image Added

Figure Delft-FEWS Archive display; search for events (filtered by flood event using double click)

Search and Download archive data

The Deltares archive will contain data that is not always included in an archive event file. If the data is not included in an event file it can be downloaded in the Search in archive tab.

Figure Delft-FEWS Archive display; search in archive

To search for data in the archive; take the following steps:1        

1. Select the area

...

2. Set the search start and end times.

2         Press the search button on the right part of the display to check if there is any data in the archive for the selected area and period.

...

1. Select the type of dataset
2. Select the source of the data (if available, from 2015.01 onwards only)
3. Press the search button
4. Select one or multiple data

...

1. sets from the list

...

1. for download

1. Press the download and import data button to download the data from the archive server and import into the local datastore. If do not want the data imported, you can also press the download (only) button.
2. Whatif template name is only active if dataset type is simulated forecasting.  This is a dropdown window will all whatif templates configured. You can filter your search results by choosing an id.
3. Whatif  senarion name is only active if dataset type is simulated forecasting. You can filter your search results by typing the name of the scenario you are searching for.
4.  Data search property windows are only available if they are configured (see example below.) The dopdowns are only active if dataset type is simulated forecasting.  You can configure no more than two search properties. 

Image Added

Figure Delft-FEWS Archive display; search in archive and download data to the local computer. 

Note: After selecting an area and period make sure to press the search in archive data before downloading the files from the archive server. Especially the National area can contain many external forecast data files and the size of the files can be considerable.

Importing archived data into Delft-FEWS stand alone application

Once If you did not import the data is downloaded from the archive (through events or data download); the data can be imported directly as part of the download, you can import the data in the Delft-FEWS Stand Alone system with use of the Manual Forecast display. To import downloaded data; take the following steps:

...

All data that is downloaded and available in the ArchiveDownload folder will be imported in the localdatastore of the Stand Alone system. As the imported data will be imported in the localDatastore of the Deft-FEWS Stand Alone it is advised to delete or rename the existing localdatastore folder and all data in the ArchiveDownload folder before starting a new Archive download and import session. Note that this import activity will bring back the data using the original data administration.

If you want to use the data within a different context, you may also copy the data to the normal Import folders, and setup a normal timeSeriesImport run module with idmapping to bring the data into the Delft-FEWS stand alone application registered under your timeSeriesImport run module.

Using archived configurations

These are the 3 ways you can make use of an archived configuration:

1. ArchiveDisplay “download and import” button can be used from version 2020.02.  It downloads and launches the selected config as a new SA application.
2. Menu item "Load latest archive configuration" has two steps : it downloads automatically the latest archived config and then replaces the current config with that downloaded latest config.
3. You can use a config downloaded through the ArchiveDisplay in your current SA config, by closing that SA and replacing the config manually.

...

1. There is currently no way to do this automatically like with the latest config (see previous item)

Archive Database

A mongo database plugin can be added to the archives to store scalar data. This tab is only available if you added a <archiveDatabase> to your config. To search for or download data from the database, different filters can (or must be used). The list of areas available is loaded from the configuration.

Image Added

When the user selects both an area and a time series type, the list of sources available will be populated. When a source is selected, the available time steps, parameters and module instance ids will be loaded from the database.

Image Added

To search for data sets, area, time series type and source need to be selected. You can also filter the results by time steps, parameters and module instance ids, but these fields aren’t mandatory: if nothing is selected, data for all of the available options will be included in the search result. To search for data, klick the magnifying glass on the left side of the panel.

Image Added

The summary of the available data sets will be displayed in the text field at the bottom of the panel.

Image Added

You can download the selected data by clicking the download button (under the search button). When the download is complete, the summary will be logged and also displayed in the panel.

Image Added

When you are downloading data from the search and download datasets panel, if part or all of the time series are stored in the mongo database, it will be automatically downloaded along with the data stored in the open archive.

Image Added

Config example:

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>url</catalogueUrl>
	<elasticSearchUrl>url</elasticSearchUrl>
	<archiveDatabase>
		<archiveDatabaseUrl>url</archiveDatabaseUrl>
		<user>user</user>
		<password>pass</password>
		<pluginClass>nl.fews.archivedatabase.mongodb.MongoDbArchiveDatabase</pluginClass>
		<pluginFactoryMethod>create</pluginFactoryMethod>
		<openDatabasePluginPanelBinDir>d:\mongdb-plugin\</openDatabasePluginPanelBinDir>
	</archiveDatabase>
</archives>

...