Introduction

Currently, the graphical user interface to access the Deltares Open Archive is via the Delft-FEWS GUI. In addition, data can be approached via the Geonetwork web client of via THREDDS web server.

The Deltares archive is located on a separate server and can be used for post-event analysis like (re-) calibration of models, reviews and performance analysis. Most of the data in the Deltares archive is archived through automated processes that are scheduled on the Delft-FEWS MC server. The archived data can be accessed and downloaded to the local computer through the Delft-FEWS Archive catalogue display.

The Deltares Archive is storing data (sets), these are the actual forecast time series, observation time series, configuration, rating curves, forecaster notes, model states, FEWS modifiers etc. Most of the data sets can be organized in events; this is a special type of metadata overarching multiple data sets. The area and period covered by the event determine, in combination with the type of event and associated data types, which data sets belong to an event.

A separate Deltares archive document is available that contains all technical details of the Deltares archive. An Installation of the Deltares Archive is available on the Deltares Delft-FEWS Wiki pages.

This user guide is derived from the HyFS application as used at the Bureau of Meteorology.

Archived data

In the Delft-FEWS application, scheduled and manual archive workflows can take care of archiving the data (sets) from the database. The scheduled archive workflow is generally run once a day and will generally archive the following type of data:

Archived data set	Data set	Data types
Astronomical data	Observed time series	NetCDF-CF timeseries
Observed series	Observed time series	NetCDF-CF timeseries
Surge forecasts	External Forecast time series	NetCDF-CF timeseries
Forecaster Notes on various level	Messages	text
Delft-FEWS configuration changes	Configuration	zip (FEWS-configuration)
Rating Curves	Rating Curves	xml (FEWS PI-rating)
NWP forecasts	External Forecast time series	NetCDF-CF timeseries
Simulations (forecast)	Simulated timeseries	NetCDF-CF timeseries Model States Modifiers Reports
Products as handed over to downstream users	Products	Products in original format
Local datastore	Snapshot	Local datastore

Note: External data which is archived elsewhere should in general not be archived within the context of the Delft-FEWS application.

Events

Within Forecasting organisations, events are situations that can be distinguished from business as ususal as something special or interesting is happening, e.g. a flood, drought or calamity. Generally the operations during an event is hectic and no one wants to spent time on data archiving. however, after the event the archived data may be crucial for legal inquiries, performance assessment, or model calibration or to use as input for training. Therefore, the concept of Event has been introduced to label a period of interest for a certain area, such that one easily can find this data back.

Many of the important data sets stored in a Deltares Archive associated with a real time system can be organized in events, this is a special type of metadata overarching one or multiple data sets. The event type, area and period covered by the event determine which data sets belong to an event. Each event type will include one or multiple data sets. Typical event types within a real time application are:

Historic Event: This event type contains observed time series that are stored in the archive. When a new historic event is made the data include in the historic event period will become available as historic event in the Delft-FEWS operator client and can be viewed together with the current observed or simulated data.
Review Event: This event type may contain observed time series, external forecast time series, simulated time series, modifiers and messages. When a new review event type is made the data sets covered will be tagged and can be downloaded to a Delft-FEWS Stand Alone for post event review purposes.
Flood Watch Event: This event type contains observed time series and external forecast time series. When a new flood watch event type is made the data sets covered will be tagged and can be downloaded to a Delft-FEWS Stand Alone for skill perfomance analysis
Watercoach/Training Event: This event type contains observed time series, external forecast time series, simulated time series, modifiers, model states and messages. When a new water coach event type is made the data sets covered will be tagged and can be downloaded to a Delft-FEWS Stand Alone. The data downloaded can then be used for making a new Delft-FEWS Water Coach script for training purposes.

Event types can be defined as appropriate, but need to be registered on the archiver server side.

Additional to tagging an event in the Deltares archive it is also possible to have attachments uploaded and linked to an event in the archive. Creating an event, downloading events and data sets, and attaching attachments to events can be done with the Delft-FEWS Archive display.

Finally, events can play a crucial role in the data management of the archive. Data labelled for a specific type of event may be help longer in the archive compared to data not labelled for an event. E.g. Numerical Weather Predictions may be removed quickly, except for those external forecast that are part of a Flood Watch or training event.

Archive display

All data that is archived can be accessed and downloaded through the Delft-FEWS archive display. The archive display can be started from the Delft-FEWS Explorer from the Tools menu.

When the display is initialized it will try to contact the archive file server to download information about the archive content (the archive configuration).

Note: When the Delft-FEWS archive display cannot connect to the archive file server, there will be a error message shown when starting the display. If this is the case, make sure the computer has a network connection, access to the archive file server, and that the path to the archive file server is correctly configured in the global.properties file.

The Delft-FEWS Archive display contains three tabs:

Create a new event: This tab can be used to create a new event file to tag data in the Deltares archive. It also contains functionality to update an event and/or upload an attachment to an existing event.
Search for events: This tab can be used to search in existing Deltares archive for events and download the associated data and the attachment(s) that have been tagged to the event.
Search in archive: This tab can be used to download data from the archive, without using the events functionality.

In the following paragraphs the functionality of the different tabs is explained.

Create an archive event

Events are created for Areas and are valid for a period; specified by the start and end time.

To create a new event; take the following steps:

1 Select the area for the event (an area is a basin or national). The list with events will be updated with all events found in the Deltares Archive.

2 Press the create event button to create a new event for the selected area.

3 Select the event type to add and enter the start and end time of the event. Complete the name and description textboxes for the event to make the new event understandable for other forecasters.

4 Press the search button on the right part of the display to check if there is any data in the archive for the selected area and period.

Location and parameter selection

The location and parameter list buttons below the search button allow you to select locations and parameters. This selection is only relevant when defining a historical event. The dataset that is created will always contain all timeserieSets. But in the case of a historical event, a list with selected parameters/locations will be included. When the dataset is imported as historical event, the locations/parameters will be used to filter the data

5 Press the save event button to save the event file on the local computer. The event will be added as a new line in the event list table in red.

6 Press the upload event button to upload the event file from the local computer to the archive server.

Figure Delft-FEWS Archive display; create new event

When an event is created it is possible to attach a file (pdf file, or zip file, max. 100Mb) to the event. Take the following steps to attach an attachment to an existing event.

1 Select the event from the list of events.

2 Press the attach file button to select a file and attach it to the selected event.

3 Press the upload event button to upload the event file from the local computer to the archive server.

Search and Download archive events

The Deltares archive will contain many events for all catchments. The events can be searched and downloaded in the Search for Events tab.

To search for events; take the following steps:

1 Select the area and set the search start and end times. The list with events will be updated with all events found in the Deltares Archive that match the selected criteria.

2 Press the search button on the right part of the display to check if there is any data in the archive for the selected area and period.

3 Select one or multiple events from the list with events.

4 Press the download attachment button to download the attachments for the selected events from the archive server to the local computer. The attachments will be stored in the ArchiveDownload folder (configured); check the log file in the system logs to find the names of the correct folders.

5 Press the download event with data button to download the data for the selected events from the archive server to the local computer.

Note: By double clicking one of the cells in the event list, the event list will be filtered and show only those events that have the same content of the selected cell. To filter for example on historic events, double click in the event type cell where the content is historic event. All cells in that event table column will now have a blue background color indicating there is a filter active. To remove the filter, double click again on one of the event type cells.

Figure Delft-FEWS Archive display; search for events (filtered by flood event using double click)

Search and Download archive data

The Deltares archive will contain data that is not always included in an archive event file. If the data is not included in an event file it can be downloaded in the Search in archive tab.

To search for data in the archive; take the following steps:

Select the area
Set the search start and end times.
Select the type of dataset
Select the source of the data (if available, from 2015.01 onwards only)
Press the search button
Select one or multiple data sets from the list for download
Press the download and import data button to download the data from the archive server and import into the local datastore. If do not want the data imported, you can also press the download (only) button.
Whatif template name is only active if dataset type is simulated forecasting. This is a dropdown window will all whatif templates configured. You can filter your search results by choosing an id.
Whatif senarion name is only active if dataset type is simulated forecasting. You can filter your search results by typing the name of the scenario you are searching for.
Data search property windows are only available if they are configured (see example below.) The dopdowns are only active if dataset type is simulated forecasting. You can configure no more than two search properties.

Figure Delft-FEWS Archive display; search in archive and download data

Note: After selecting an area and period make sure to press the search in archive data before downloading the files from the archive server. Especially the National area can contain many external forecast data files and the size of the files can be considerable.

Importing archived data into Delft-FEWS stand alone application

If you did not import the data directly as part of the download, you can import the data in the Delft-FEWS Stand Alone system with use of the Manual Forecast display. To import downloaded data; take the following steps:

Open the Manual Forecast display from the Delft-FEWS Explorer Menu or Toolbar.
Select the Import data from Archive workflow.
Press the Run button.

All data that is downloaded and available in the ArchiveDownload folder will be imported in the localdatastore of the Stand Alone system. As the imported data will be imported in the localDatastore of the Deft-FEWS Stand Alone it is advised to delete or rename the existing localdatastore folder and all data in the ArchiveDownload folder before starting a new Archive download and import session. Note that this import activity will bring back the data using the original data administration.

If you want to use the data within a different context, you may also copy the data to the normal Import folders, and setup a normal timeSeriesImport run module with idmapping to bring the data into the Delft-FEWS stand alone application registered under your timeSeriesImport run module.

Behavior of imported historical simulations

When multiple historical simulations are imported from the archive, it is not possible to switch between these simulations in a similar manner as can be done in the original system, by making a previous simulation “Current". When importing multiple historical simulations, each newly imported simulation will overwrite the results of a previous simulation, similar to normal observations.

Therefore, only import the historical simulations that are really needed. When the import is needed to retrieve the warm state that was used to create a forecast, only import that historical simulation. If the intention is to rerun a historical simulation, first import the historical simulation that provided the warm state for the historical simulation of interest. Then import that historical simulation itself.

Using archived configurations

These are the 3 ways you can make use of an archived configuration:

ArchiveDisplay “download and import” button can be used from version 2020.02. It downloads and launches the selected config as a new SA application.
Menu item "Load latest archive configuration" has two steps : it downloads automatically the latest archived config and then replaces the current config with that downloaded latest config.
You can use a config downloaded through the ArchiveDisplay in your current SA config, by closing that SA and replacing the config manually. There is currently no way to do this automatically like with the latest config (see previous item)

Config example:

hideDownloadDataSetsTab, hideCreateEventTab, hideSearchEventTab, hideArchiveDatabaseTab, hideArchiveUploadTab : Available since 2022.01. Optional fields, default value is false. If it’s set to true, the tab in question will never be visible in the Archive.

archiveUploadIdMapId: Since 2022.02. Optional. Id of the id mapping to be used in the archive upload panel.

initialSearchTimePeriod: Since 2022.01 Optional. if configured the search window in the archive catalogue will be set to this period upon starting up.

dataSearchProperty: Since 2022.01. Optional. If configured, a dropdown box with the possible values of the property will be added to the display as a possible filter for simulated time series. No more than two search properites can be added.

<?xml version="1.0" encoding="UTF-8"?>
<archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
					  xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archiveModuleDisplay.xsd">
	<levelThresholds>
		<levelThresholdId>MIN</levelThresholdId>
	</levelThresholds>
	<workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/work</workFolder>
	<createEventPermission>Admin</createEventPermission>
	<deleteEventPermission>Admin</deleteEventPermission>
	<disableDataDownloadAtOperatorClient>true</disableDataDownloadAtOperatorClient>
	<downloadFolders>
		<downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/Observed</downloadFolderObserved>
		<downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/Simulated</downloadFolderSimulated>
		<downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/ExternalForecasts</downloadFolderExternalForecasts>
		<downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/RatingCurves</downloadFolderRatingCurves>
		<downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/HistoricalEvents</downloadFolderHistoricalEvents>
		<downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/ForecasterNotes</downloadFolderForecasterNotes>
		<downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/Configuration</downloadFolderConfiguration>
		<downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/Attachments</downloadFolderEventAttachments>
		<downloadFolderSnapShots>$ARCHIVE_DOWNLOAD_FOLDER$/SnapShots</downloadFolderSnapShots>
		<downloadFolderProducts>$ARCHIVE_DOWNLOAD_FOLDER$/Products</downloadFolderProducts>
	</downloadFolders>
	<archiveImportWorkflowId>Archive_Import</archiveImportWorkflowId>
	<hideDownloadDataSetsTab>true</hideDownloadDataSetsTab>
	<hideCreateEventTab>true</hideCreateEventTab>
	<hideSearchEventTab>true</hideSearchEventTab>
	<hideArchiveDatabaseTab>true</hideArchiveDatabaseTab>
	<hideArchiveUploadTab>true</hideArchiveUploadTab>
	<archiveUploadIdMapId>id</archiveUploadIdMapId>
	<initialSearchTimePeriod start="-1000" end="1" unit="day"></initialSearchTimePeriod>
	<dataSearchProperty>
		<propertyId>aVeryLongPropertyId</propertyId>
		<displayName>propertyA</displayName>
	</dataSearchProperty>
	<dataSearchProperty>
		<propertyId>propertyB</propertyId>
	</dataSearchProperty>
</archiveModuleDisplay>

Archive Database

A mongo database plugin can be added to the archives to store scalar data. This tab is only available if you added a <archiveDatabase> to your config. To search for or download data from the database, different filters can (or must be used). The list of areas available is loaded from the configuration.

When the user selects both an area and a time series type, the list of sources available will be populated. When a source is selected, the available time steps, parameters and module instance ids will be loaded from the database.

To search for data sets, area, time series type and source need to be selected. You can also filter the results by time steps, parameters and module instance ids, but these fields aren’t mandatory: if nothing is selected, data for all of the available options will be included in the search result. To search for data, klick the magnifying glass on the left side of the panel.

The summary of the available data sets will be displayed in the text field at the bottom of the panel.

You can download the selected data by clicking the download button (under the search button). When the download is complete, the summary will be logged and also displayed in the panel.

When you are downloading data from the search and download datasets panel, if part or all of the time series are stored in the mongo database, it will be automatically downloaded along with the data stored in the open archive.