| Table of Contents |
|---|
Overview
The archive display can be used to interact with the FEWS-archive.
The main functions which are supported by this display are:
- management of events in the archive,
- search for events and download data related to events from the archive,
- search for data in the archive
It can be used to search for data, download data and import data from the archive.
Configuration
Note that when you start using the archive you need to configure the Archives.xml Archives.xml - 2018.02 and later - DELFT-FEWS Documentation - Deltares Public Wiki
...
To use the archive display, configure it as <explorerTask> in Explorer.xml. For example:
| Code Block | ||||
|---|---|---|---|---|
| ||||
<explorerTask name="Archive Catalogue"> <iconFile>Archive.png</iconFile> <mnemonic>A</mnemonic> <displayConfigFileName>ArchiveModuleDisplay</displayConfigFileName> <toolbarTask>false</toolbarTask> <menubarTask>true</menubarTask> <accelerator>ctrl A</accelerator> <loadAtStartup>false</loadAtStartup> </explorerTask> |
The ArchiveModuleDisplay configuration
...
The archive display uses the file Archives.xml file in the SystemConfigFiles directory to identify were the archive web server is located.
The Archives.xml is explained in more detail in the section seamless integration.
...
Below an example is shown.
...
<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd">
<catalogueUrl>http://servername:portname/deltares-archive-server/catalogue</catalogueUrl>
</archives>
If the archive is installed following the conventions described in Installation of the Deltares OpenArchive (earlier versions) the configuration shown above is sufficient.
Note however that when you start using the seamless integration in the pi-webservice that more configuration is required in the Archives.xml file.
The archives.xml will be used by the archive display panel to identify were the webservices of the archive are located.
In addition you will need to configure a file called ArchiveModuleDisplay.xml in the DisplayConfigFiles folder.
Below an example of this file containing the most important tags is shown.
| Code Block | ||||
|---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="UTF-8"?> <archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/archiveModuleDisplay.xsd"> <workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/workdir</workFolder> <downloadFolders> <downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/observed</downloadFolderObserved> <downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/simulated</downloadFolderSimulated> <downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/externalforecasts</downloadFolderExternalForecasts> <downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/ratingcurves</downloadFolderRatingCurves> <downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/historicalevents</downloadFolderHistoricalEvents> <downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/messages</downloadFolderForecasterNotes> <downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/configuration</downloadFolderConfiguration> <downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/attachments</downloadFolderEventAttachments> </downloadFolders> <archiveImportWorkflowId>ArchiveImport</archiveImportWorkflowId> </archiveModuleDisplay> |
The $ARCHIVE_DOWNLOAD_FOLDER$ should be configured in your global.properties file and should point to the folder were the panel should download the files of the archive to.
...
downloads the data first and runs an import workflow directly after finishing the download. The id configured in the tag archiveImportWorkflowId identifies which workflow should be ran.
Below a more advanced example of a ArchiveModuleDisplay.xml is shown.
.
Additional configuration options
hideDownloadDataSetsTab, hideCreateEventTab, hideSearchEventTab, hideArchiveDatabaseTab, hideArchiveUploadTab : Available since 2022.01. Optional fields, default value is false. If it’s set to true, the tab in question will never be visible in the Archive.
archiveUploadIdMapId: Since 2022.02. Optional. Id of the id mapping to be used in the archive upload panel.
initialSearchTimePeriod: Since 2022.01 Optional. if configured the search window in the archive catalogue will be set to this period upon starting up.
dataSearchProperty: Since 2022.01. Optional. If configured, a dropdown box with the possible values of the property will be added to the display as a possible filter for simulated time series. No more than two search properites can be added.| Code Block | ||||
|---|---|---|---|---|
| ||||
| Code Block | ||||
| ||||
<?xml version="1.0" encoding="UTF-8"?> <archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/archiveModuleDisplay.xsd"> <levelThresholds> <levelThresholdId>threshold id< <levelThresholdId>MIN</levelThresholdId> <levelThresholdId>second threshold id</levelThresholdId> </levelThresholds> <defaultHistoricalEventTimeStep unit="hour"></defaultHistoricalEventTimeStep> <workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/workdir<work</workFolder> <createEventPermission>create permission<<createEventPermission>Admin</createEventPermission> <deleteEventPermission>delete permisson<<deleteEventPermission>Admin</deleteEventPermission> <disableDataDownloadAtOperatorClient>true</disableDataDownloadAtOperatorClient> <downloadFolders> <downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/observed<Observed</downloadFolderObserved> <downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/simulated<Simulated</downloadFolderSimulated> <downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/externalforecasts<ExternalForecasts</downloadFolderExternalForecasts> <downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/ratingcurves<RatingCurves</downloadFolderRatingCurves> <downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/historicalevents<HistoricalEvents</downloadFolderHistoricalEvents> <downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/messages<ForecasterNotes</downloadFolderForecasterNotes> <downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/configuration<Configuration</downloadFolderConfiguration> <downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/attachments<Attachments</downloadFolderEventAttachments> <downloadFolderSnapShots>$ARCHIVE_DOWNLOAD_FOLDER$/SnapShots</downloadFolderSnapShots> <downloadFolderProducts>$ARCHIVE_DOWNLOAD_FOLDER$/Products</downloadFolderProducts> </downloadFolders> <archiveImportWorkflowId>ArchiveImport</archiveImportWorkflowId> <dataTypeImportWorkflows> <statesWorkflowId>ImportArchivedStates</statesWorkflowId> </dataTypeImportWorkflows> <searchDataTypes> <type>simulated</type> <type>observed</type> </searchDataTypes> </archiveModuleDisplay> |
<archiveImportWorkflowId>Archive_Import</archiveImportWorkflowId>
<hideDownloadDataSetsTab>true</hideDownloadDataSetsTab>
<hideCreateEventTab>true</hideCreateEventTab>
<hideSearchEventTab>true</hideSearchEventTab>
<hideArchiveDatabaseTab>true</hideArchiveDatabaseTab>
<hideArchiveUploadTab>true</hideArchiveUploadTab>
<archiveUploadIdMapId>id</archiveUploadIdMapId>
<initialSearchTimePeriod start="-1000" end="1" unit="day"></initialSearchTimePeriod>
<dataSearchProperty>
<propertyId>aVeryLongPropertyId</propertyId>
<displayName>propertyA</displayName>
</dataSearchProperty>
<dataSearchProperty>
<propertyId>propertyB</propertyId>
</dataSearchProperty>
</archiveModuleDisplay>
|
Archive panel
The archive display has several tabs which are shown in the picture below.
The functionality of each tab is explained in the paragraphs below:
Search and download datasets
This The events- panel can be used to search for events specific datasets in the archive. It is possible to search for events in which certain thresholds were crossed.
The tag levelThresholds can be used to configure for which thresholds can be searched in the eventspanel.
By default every user can delete or create events. The tags createEventPermission and deleteEventPermission can be used to restrict those actions to users which have the required permissions.
The "Search and download datasets"-panel can be used to search for data in the archive. The dropdown-box in this panel shows by default all possible datasets-types.
To restrict this list, for example when not all datasets-types are exported by FEWS to the archive, the tag searchDataTypes can be used.
...
by area, source, period and data set type.
After selecting one or more data sets it is possible to download the selected data sets by pressing the download button 
. Importing the data is possible by using the import button 
Importing grids as reference (available since 2023.01)
Downloading and importing grid data can be time consuming, and the data can take up much space in the database. If one is connected to the archive, it is also possible to only import a reference to the data by selecting the tick box 'import grids as reference.' This requires the user, however, to maintain access to that file for the data to be used later. This option is available for the 'search and download datasets' and 'search and download events' panel. Once the user decides to import the grid data, the data will not be downloaded but only a reference to where the data can be found in the archive is stored. When the data is needed in the FEWS system, for example when the user wants to visualize the data or use it in a workflow, this data is downloaded (streamed) by using the stored reference. The data will be stored in the local cache and is not stored in the database.
N.B. The actual download and import process works as follows. Upon pressing the 'download data' or 'download and import data' button, FEWS will download the relevant metadata.xml and runinfo.xml files from the archive, and place a dummy (0 byte) netCDF file next to it. In the import process, FEWS will recognize that the data is a dummy, and then tries to import the reference by asking the archive for the correct path to the data.
Importing states
The states can be downloaded and imported as a part of ‘simulated forecast’ data set, together with simulated time series, reports and modifiers. For this purpose an item ‘simulated forecast’ should be selected from the data set drop down list.
...
When ‘states’ is selected, all simulated data sets that include states are listed in the table and the ‘Download and import data’ button becomes enabled. To download and import the states, select one or more data sets from the table and press button ‘Download and import data’ . The states data sets will be first downloaded to the archive download folder ./simulated and then imported in the database table WarmStates. If there is already a state for the same module and with the same state time in the database table, the downloaded state will be skipped.
Custom configuration
If the archive is not installed according to the conventions desribed in the installation manual than it might be needed to configure custom endpoint for the archive-webservices which
are used by the archive panel. This custom configuration can be done in the tag "archiveServer", below an example is given.
...
Create a new event
This panel can be used to create, update and delete events. The content of the selected event can be examined by using the 
button in the summary of archive data panel.
The events panel can be used to search for events in the archive. It is possible to search for events in which certain thresholds were crossed. The tag levelThresholds can be used to configure for which thresholds can be searched in the events panel. By default every user can delete or create events.
Permissions
The tags createEventPermission and deleteEventPermission can be used to restrict those actions to users which have the required permissions.
The "Search and download datasets"-panel can be used to search for data in the archive. The dropdown-box in this panel shows by default all possible datasets-types.
To restrict this list, for example when not all datasets-types are exported by FEWS to the archive, the tag searchDataTypes can be used.
Search and download events
This panel can be used to download and/or import data from existing events. Since 2023.01 it is also possible to import grids as reference only (see Section 'importing grids as reference' above).
Archive Database
A mongo database plugin can be added to the archives to store scalar data. This tab is only available if you added a <archiveDatabase> to your config. To search for or download data from the database, different filters can (or must be used). The list of areas available is loaded from the configuration.
When the user selects both an area and a time series type, the list of sources available will be populated. When a source is selected, the available time steps, parameters and module instance ids will be loaded from the database.
To search for data sets, area, time series type and source need to be selected. You can also filter the results by time steps, parameters and module instance ids, but these fields aren’t mandatory: if nothing is selected, data for all of the available options will be included in the search result. To search for data, klick the magnifying glass on the left side of the panel.
The summary of the available data sets will be displayed in the text field at the bottom of the panel.
You can download the selected data by clicking the download button (under the search button). When the download is complete, the summary will be logged and also displayed in the panel.
When you are downloading data from the search and download datasets panel, if part or all of the time series are stored in the mongo database, it will be automatically downloaded along with the data stored in the open archive.
Config example:
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
<catalogueUrl>url</catalogueUrl>
<elasticSearchUrl>url</elasticSearchUrl>
<archiveDatabase>
<archiveDatabaseUrl>url</archiveDatabaseUrl>
<user>user</user>
<password>pass</password>
<pluginClass>nl.fews.archivedatabase.mongodb.MongoDbArchiveDatabase</pluginClass>
<pluginFactoryMethod>create</pluginFactoryMethod>
<openDatabasePluginPanelBinDir>d:\mongdb-plugin\</openDatabasePluginPanelBinDir>
</archiveDatabase>
</archives> |
Upload to archive
It is possible to add quality controlled external historical data to your existing archive. This can be done by using the tab "Upload to archive".
It is possible to define permission for this tab. This can be done adding the tag <uploadNewDataPermission>uploadNewDataPermission</uploadNewDataPermission> to the archiveModuleDisplay. The "Upload to archive" tab will be hidden for users who don't have the correct permissions.
The following steps are usually taken when adding quality controlled data to your archive.
1) Import the data you want to add in your stand-alone FEWS application,
2) Verify the data by checking it in the Plots display and edit the data where needed.
3) When the data is verified and approved then you can upload the data to your archive by using the archive display. A new tab (available from 2022.02) Upload to Archive is available. Start the archive display and select the Upload to Archive tab.
4) Select the time series you want to upload by using the data viewer. You can select the period you want to upload by selecting the period in the panel. In addition you should select the area to which you want to add the data.
5) Before you can upload the data you should press the "Check availability in catalog" button. After pressing the button the display will check for which periods the selected time series are already available in the archive. In addition it will check which time series are edited in the periods for which the data is already archived. In the example below there is already data available in the archive for a part of the selected period. The last selected time series also has 2 data edits in the period for which the data is already archived.
6) After step 5 the button "Upload to archive" will enabled. Note that when you change the selected time series the button will be unavailable again and that you should check the availability of the data again. When you press the button "Upload to archive" the selected time series will be uploaded to archive. For the periods for which the time series are not already archived to time series will be exported to netcdf and uploaded to the archive. For the periods for which the data is already archived only the edited values will be uploaded in pi-xml format to the archive. Both uploads are handled by the webservices of the archive. To insert the edited data into the archive a workflow in FEWS must be executed. To insert the newly added netcdf-files into the archive a task at the archive server must be executed. The upload process will determine automatically to which areas the time series belong. This area is determined by the value of a configured location attribute id. It is possible to configure more than one location attribute id. If a location belongs to multiple areas the new data will be added to all areas.