...
The archive display can be used to interact with the FEWS-archive. It can be used to search for data, download data and import data from the archive.
Configuration
Note that when you start using the archive you need to configure the Archives.xml Archives.xml - 2018.02 and later - DELFT-FEWS Documentation - Deltares Public Wiki
To use the archive display, configure it as <explorerTask> in Explorer.xml. For example:
...
The ArchiveModuleDisplay configuration
The archive display uses the file Archives.xml file in the SystemConfigFiles directory to identify were the archive web server is located. The Archives.xml is explained in more detail in the section seamless integration. Below an example is shown.
...
<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd">
<catalogueUrl>http://dummy_server:dummy_port/deltares-archive-server/catalogue</catalogueUrl>
</archives>Archives.xml
...
Below an example of this file containing the most important tags is shown.
| Code Block | ||||
|---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="UTF-8"?> <archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/archiveModuleDisplay.xsd"> <workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/workdir</workFolder> <downloadFolders> <downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/observed</downloadFolderObserved> <downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/simulated</downloadFolderSimulated> <downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/externalforecasts</downloadFolderExternalForecasts> <downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/ratingcurves</downloadFolderRatingCurves> <downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/historicalevents</downloadFolderHistoricalEvents> <downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/messages</downloadFolderForecasterNotes> <downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/configuration</downloadFolderConfiguration> <downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/attachments</downloadFolderEventAttachments> </downloadFolders> <archiveImportWorkflowId>ArchiveImport</archiveImportWorkflowId> </archiveModuleDisplay> |
...
downloads the data first and runs an import workflow directly after finishing the download. The id configured in the tag archiveImportWorkflowId identifies which workflow should be ran.
...
Additional configuration options
If the archive is not installed according to the conventions described in the installation manual than it might be needed to configure custom endpoint for the archive-webservices which are used by the archive panel. This custom configuration can be done in the tag "archiveServer", below an example is given.
...
<archiveServer>
<archiveWebService>http://dummy_server:8080/deltares-archive-server</archiveWebService>
<archiveConfigFileUrl>http://dummy_server:8080/thredds/fileServer/Config</archiveConfigFileUrl>
<eventsFileServerUrl>http://dummy_server:8080/thredds/fileServer/Events</eventsFileServerUrl>
<dataFileServerUrl>http://dummy_server:8080/thredds/fileServer/archive</dataFileServerUrl>
</archiveServer>hideDownloadDataSetsTab, hideCreateEventTab, hideSearchEventTab, hideArchiveDatabaseTab, hideArchiveUploadTab : Available since 2022.01. Optional fields, default value is false. If it’s set to true, the tab in question will never be visible in the Archive.
archiveUploadIdMapId: Since 2022.02. Optional. Id of the id mapping to be used in the archive upload panel.
initialSearchTimePeriod: Since 2022.01 Optional. if configured the search window in the archive catalogue will be set to this period upon starting up.
dataSearchProperty: Since 2022.01. Optional. If configured, a dropdown box with the possible values of the property will be added to the display as a possible filter for simulated time series. No more than two search properites can be added.| Code Block | ||||
|---|---|---|---|---|
|
Below a more advanced example of a ArchiveModuleDisplay.xml is shown.
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?> <archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/archiveModuleDisplay.xsd"> <levelThresholds> <levelThresholdId>threshold id<<levelThresholdId>MIN</levelThresholdId> <levelThresholdId>second threshold id</levelThresholdId> </levelThresholds> <defaultHistoricalEventTimeStep unit="hour"></defaultHistoricalEventTimeStep> <workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/workdir<work</workFolder> <createEventPermission>create permission<<createEventPermission>Admin</createEventPermission> <deleteEventPermission>delete permisson<<deleteEventPermission>Admin</deleteEventPermission> <disableDataDownloadAtOperatorClient>true</disableDataDownloadAtOperatorClient> <downloadFolders> <downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/observed<Observed</downloadFolderObserved> <downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/simulated<Simulated</downloadFolderSimulated> <downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/externalforecasts<ExternalForecasts</downloadFolderExternalForecasts> <downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/ratingcurves<RatingCurves</downloadFolderRatingCurves> <downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/historicalevents<HistoricalEvents</downloadFolderHistoricalEvents> <downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/messages<ForecasterNotes</downloadFolderForecasterNotes> <downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/configuration<Configuration</downloadFolderConfiguration> <downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/attachments<Attachments</downloadFolderEventAttachments> <downloadFolderSnapShots>$ARCHIVE_DOWNLOAD_FOLDER$/SnapShots</downloadFolderSnapShots> <downloadFolderProducts>$ARCHIVE_DOWNLOAD_FOLDER$/Products</downloadFolderProducts> </downloadFolders> <archiveImportWorkflowId>ArchiveImport<<archiveImportWorkflowId>Archive_Import</archiveImportWorkflowId> <dataTypeImportWorkflows> <statesWorkflowId>ImportArchivedStates</statesWorkflowId> </dataTypeImportWorkflows> <searchDataTypes> <type>simulated</type> <type>observed</type> </searchDataTypes> </archiveModuleDisplay> <hideDownloadDataSetsTab>true</hideDownloadDataSetsTab> <hideCreateEventTab>true</hideCreateEventTab> <hideSearchEventTab>true</hideSearchEventTab> <hideArchiveDatabaseTab>true</hideArchiveDatabaseTab> <hideArchiveUploadTab>true</hideArchiveUploadTab> <archiveUploadIdMapId>id</archiveUploadIdMapId> <initialSearchTimePeriod start="-1000" end="1" unit="day"></initialSearchTimePeriod> <dataSearchProperty> <propertyId>aVeryLongPropertyId</propertyId> <displayName>propertyA</displayName> </dataSearchProperty> <dataSearchProperty> <propertyId>propertyB</propertyId> </dataSearchProperty> </archiveModuleDisplay> |
Archive panel
The archive display has several tabs which as are shown in the picture below.
...
Search and download datasets
...
The events This panel can be used to search for events specific datasets in the archive. It is possible to search for events in which certain thresholds were crossed. The tag levelThresholds can be used to configure for which thresholds can be searched in the events panel. By default every user can delete or create events.
Permissions
The tags createEventPermission and deleteEventPermission can be used to restrict those actions to users which have the required permissions.
The "Search and download datasets"-panel can be used to search for data in the archive. The dropdown-box in this panel shows by default all possible datasets-types.
by area, source, period and data set type.
After selecting one or more data sets it is possible to download the selected data sets by pressing the download button 
. Importing the data is possible by using the import button 
Importing grids as reference (available since 2023.01)
Downloading and importing grid data can be time consuming, and the data can take up much space in the database. If one is connected to the archive, it is also possible to only import a reference to the data by selecting the tick box 'import grids as reference.' This requires the user, however, to maintain access to that file for the data to be used later. This option is available for the 'search and download datasets' and 'search and download events' panel. Once the user decides to import the grid data, the data will not be downloaded but only a reference to where the data can be found in the archive is stored. When the data is needed in the FEWS system, for example when the user wants to visualize the data or use it in a workflow, this data is downloaded (streamed) by using the stored reference. The data will be stored in the local cache and is not stored in the database.
N.B. The actual download and import process works as follows. Upon pressing the 'download data' or 'download and import data' button, FEWS will download the relevant metadata.xml and runinfo.xml files from the archive, and place a dummy (0 byte) netCDF file next to it. In the import process, FEWS will recognize that the data is a dummy, and then tries to import the reference by asking the archive for the correct path to the dataTo restrict this list, for example when not all datasets-types are exported by FEWS to the archive, the tag searchDataTypes can be used.
Importing states
The states can be downloaded and imported as a part of ‘simulated forecast’ data set, together with simulated time series, reports and modifiers. For this purpose an item ‘simulated forecast’ should be selected from the data set drop down list.
...
When ‘states’ is selected, all simulated data sets that include states are listed in the table and the ‘Download and import data’ button becomes enabled. To download and import the states, select one or more data sets from the table and press button ‘Download and import data’.The states data sets will be first downloaded to the archive download folder ./simulated and then imported in the database table WarmStates. If there is already a state for the same module and with the same state time in the database table, the downloaded state will be skipped.
Config example:
hideDownloadDataSetsTab, hideCreateEventTab, hideSearchEventTab, hideArchiveDatabaseTab, hideArchiveUploadTab : Available since 2022.01. Optional fields, default value is false. If it’s set to true, the tab in question will never be visible in the Archive.
archiveUploadIdMapId: Since 2022.02. Optional. Id of the id mapping to be used in the archive upload panel.
initialSearchTimePeriod: Since 2022.01 Optional. if configured the search window in the archive catalogue will be set to this period upon starting up.
dataSearchProperty: Since 2022.01. Optional. If configured, a dropdown box with the possible values of the property will be added to the display as a possible filter for simulated time series. No more than two search properites can be added....
| language | xml |
|---|---|
| linenumbers | true |
...
Create a new event
This panel can be used to create, update and delete events. The content of the selected event can be examined by using the 
button in the summary of archive data panel.
The events panel can be used to search for events in the archive. It is possible to search for events in which certain thresholds were crossed. The tag levelThresholds can be used to configure for which thresholds can be searched in the events panel. By default every user can delete or create events.
Permissions
The tags createEventPermission and deleteEventPermission can be used to restrict those actions to users which have the required permissions.
The "Search and download datasets"-panel can be used to search for data in the archive. The dropdown-box in this panel shows by default all possible datasets-types.
To restrict this list, for example when not all datasets-types are exported by FEWS to the archive, the tag searchDataTypes can be used.
Search and download events
This panel can be used to download and/or import data from existing events. Since 2023.01 it is also possible to import grids as reference only (see Section 'importing grids as reference' above).
Archive Database
A mongo database plugin can be added to the archives to store scalar data. This tab is only available if you added a <archiveDatabase> to your config. To search for or download data from the database, different filters can (or must be used). The list of areas available is loaded from the configuration.
...
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews httphttps://fewsfewsdocs.wldelftdeltares.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>url</catalogueUrl> <elasticSearchUrl>url</elasticSearchUrl> <archiveDatabase> <archiveDatabaseUrl>url</archiveDatabaseUrl> <user>user</user> <password>pass</password> <pluginClass>nl.fews.archivedatabase.mongodb.MongoDbArchiveDatabase</pluginClass> <pluginFactoryMethod>create</pluginFactoryMethod> <openDatabasePluginPanelBinDir>d:\mongdb-plugin\</openDatabasePluginPanelBinDir> </archiveDatabase> </archives> |
...
It is possible to add quality controlled external historical data to you your existing archive. This can be done by using the tab "Upload to archive".
It is possible to define permission for this tab. This can be done adding the tag <uploadNewDataPermission>uploadNewDataPermission</uploadNewDataPermission> to the archiveModuleDisplay. The "Upload to archive" tab will be hidden for users who don't have the following procedurecorrect permissions.
The following steps are usually taken when adding quality controlled data to your archive.
1) Import the data you want to add in your stand-alone FEWS application,
...
3) When the data is verified and approved then you can upload the data to your archive by using the archive display. A new tab (available from 2022.02) Upload to Archive is available. Start the achive archive display and select the Upload to Archive tab.
...
5) Before you can upload the data you should press the "Check availability in catalog" button. After pressing the button the display will check for which periods the selected time series are already available in the archive. In addition it will check which time series are edited in the periods for which the data is already archived. In the example below there is already data available in the archive for a part of the selected period. The last selected time series also has 2 data edits in the period for which the data is already archived.
6) After step 5 the button "Upload to archive" will enabled. Note that when you change the selected time series the button will be unavailable again and that you should check the availability of the data again. When you press the button "Upload to archive" the selected time series will be uploaded to archive. For the periods for which the time series are not already archived to time series will be exported to netcdf and uploaded to the archive. For the periods for which the data is already archived only the edited values will be uploaded in pi-xml format to the archive. Both uploads are handled by the webservices of the archive. To insert the edited data into the archive a workflow in FEWS must be executed. To insert the newly added netcdf-files into the archive a task at the archive server must be executed. The upload process will determine automatically to which areas the time series belong. This area is determined by the value of a configured location attribute id. It is possible to configure more than one location attribute id. If a location belongs to multiple areas the new data will be added to all areas.