Overview
The archive display can be used to interact with the FEWS-archive. It can be used to search for data, download data and import data from the archive.
Configuration
To use the archive display, configure it as <explorerTask> in Explorer.xml. For example:
<explorerTask name="Archive Catalogue"> <iconFile>Archive.png</iconFile> <mnemonic>A</mnemonic> <displayConfigFileName>ArchiveModuleDisplay</displayConfigFileName> <toolbarTask>false</toolbarTask> <menubarTask>true</menubarTask> <accelerator>ctrl A</accelerator> <loadAtStartup>false</loadAtStartup> </explorerTask>
The ArchiveModuleDisplay configuration
The archive display uses the file Archives.xml file in the SystemConfigFiles directory to identify were the archive web server is located. The Archives.xml is explained in more detail in the section seamless integration. Below an example is shown.
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>http://dummy_server:dummy_port/deltares-archive-server/catalogue</catalogueUrl> </archives>
Archives.xml
Note that when you start using the seamless integration in the pi-webservice that more configuration is required in the Archives.xml file. The archives.xml will be used by the archive display panel to identify were the webservices of the archive are located. In addition you will need to configure a file called ArchiveModuleDisplay.xml in the DisplayConfigFiles folder.
Below an example of this file containing the most important tags is shown.
<?xml version="1.0" encoding="UTF-8"?> <archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archiveModuleDisplay.xsd"> <workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/workdir</workFolder> <downloadFolders> <downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/observed</downloadFolderObserved> <downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/simulated</downloadFolderSimulated> <downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/externalforecasts</downloadFolderExternalForecasts> <downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/ratingcurves</downloadFolderRatingCurves> <downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/historicalevents</downloadFolderHistoricalEvents> <downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/messages</downloadFolderForecasterNotes> <downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/configuration</downloadFolderConfiguration> <downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/attachments</downloadFolderEventAttachments> </downloadFolders> <archiveImportWorkflowId>ArchiveImport</archiveImportWorkflowId> </archiveModuleDisplay>
The $ARCHIVE_DOWNLOAD_FOLDER$ should be configured in your global.properties file and should point to the folder were the panel should download the files of the archive to.
The tag workFolder configures which folder will be used as the workfolder. The workfolder is used by the event-panels.
The archiveImportWorkflowId is used to configure the id of the import workflow. The panel offers two options for downloading data. The first option only downloads the data. The second option
downloads the data first and runs an import workflow directly after finishing the download. The id configured in the tag archiveImportWorkflowId identifies which workflow should be ran.
Below a more advanced example of a ArchiveModuleDisplay.xml is shown.
<?xml version="1.0" encoding="UTF-8"?>
<archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archiveModuleDisplay.xsd">
<levelThresholds>
<levelThresholdId>threshold id</levelThresholdId>
<levelThresholdId>second threshold id</levelThresholdId>
</levelThresholds>
<defaultHistoricalEventTimeStep unit="hour"></defaultHistoricalEventTimeStep>
<workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/workdir</workFolder>
<createEventPermission>create permission</createEventPermission>
<deleteEventPermission>delete permisson</deleteEventPermission>
<disableDataDownloadAtOperatorClient>true</disableDataDownloadAtOperatorClient>
<downloadFolders>
<downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/observed</downloadFolderObserved>
<downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/simulated</downloadFolderSimulated>
<downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/externalforecasts</downloadFolderExternalForecasts>
<downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/ratingcurves</downloadFolderRatingCurves>
<downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/historicalevents</downloadFolderHistoricalEvents>
<downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/messages</downloadFolderForecasterNotes>
<downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/configuration</downloadFolderConfiguration>
<downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/attachments</downloadFolderEventAttachments>
</downloadFolders>
<archiveImportWorkflowId>ArchiveImport</archiveImportWorkflowId>
<dataTypeImportWorkflows>
<statesWorkflowId>ImportArchivedStates</statesWorkflowId>
</dataTypeImportWorkflows>
<searchDataTypes>
<type>simulated</type>
<type>observed</type>
</searchDataTypes>
</archiveModuleDisplay>
Archive panel
The archive display has several tabs which as shown in the picture below.
The functionality of each tab is explained in the paragraphs below:
Search and download datasets
Events panel
The events panel can be used to search for events in the archive. It is possible to search for events in which certain thresholds were crossed. The tag levelThresholds can be used to configure for which thresholds can be searched in the events panel. By default every user can delete or create events.
Permissions
The tags createEventPermission and deleteEventPermission can be used to restrict those actions to users which have the required permissions.
The "Search and download datasets"-panel can be used to search for data in the archive. The dropdown-box in this panel shows by default all possible datasets-types.
To restrict this list, for example when not all datasets-types are exported by FEWS to the archive, the tag searchDataTypes can be used.
Importing states
The states can be downloaded and imported as a part of ‘simulated forecast’ data set, together with simulated time series, reports and modifiers. For this purpose an item ‘simulated forecast’ should be selected from the data set drop down list.
From 2019.02 it is also possible to download and import states only. To download and import states only, select first the data set 'simulated forecast' and then ‘states’ from the drop down list behind the button 'Select...'
When ‘states’ is selected, all simulated data sets that include states are listed in the table and the ‘Download and import data’ button becomes enabled. To download and import the states, select one or more data sets from the table and press button ‘Download and import data’. The states data sets will be first downloaded to the archive download folder ./simulated and then imported in the database table WarmStates. If there is already a state for the same module and with the same state time in the database table, the downloaded state will be skipped.
Config example:
hideDownloadDataSetsTab, hideCreateEventTab, hideSearchEventTab, hideArchiveDatabaseTab, hideArchiveUploadTab : Available since 2022.01. Optional fields, default value is false. If it’s set to true, the tab in question will never be visible in the Archive.
archiveUploadIdMapId: Since 2022.02. Optional. Id of the id mapping to be used in the archive upload panel.
initialSearchTimePeriod: Since 2022.01 Optional. if configured the search window in the archive catalogue will be set to this period upon starting up.
dataSearchProperty: Since 2022.01. Optional. If configured, a dropdown box with the possible values of the property will be added to the display as a possible filter for simulated time series. No more than two search properites can be added.
<?xml version="1.0" encoding="UTF-8"?> <archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archiveModuleDisplay.xsd"> <levelThresholds> <levelThresholdId>MIN</levelThresholdId> </levelThresholds> <workFolder>$ARCHIVE_DOWNLOAD_FOLDER$/work</workFolder> <createEventPermission>Admin</createEventPermission> <deleteEventPermission>Admin</deleteEventPermission> <disableDataDownloadAtOperatorClient>true</disableDataDownloadAtOperatorClient> <downloadFolders> <downloadFolderObserved>$ARCHIVE_DOWNLOAD_FOLDER$/Observed</downloadFolderObserved> <downloadFolderSimulated>$ARCHIVE_DOWNLOAD_FOLDER$/Simulated</downloadFolderSimulated> <downloadFolderExternalForecasts>$ARCHIVE_DOWNLOAD_FOLDER$/ExternalForecasts</downloadFolderExternalForecasts> <downloadFolderRatingCurves>$ARCHIVE_DOWNLOAD_FOLDER$/RatingCurves</downloadFolderRatingCurves> <downloadFolderHistoricalEvents>$ARCHIVE_DOWNLOAD_FOLDER$/HistoricalEvents</downloadFolderHistoricalEvents> <downloadFolderForecasterNotes>$ARCHIVE_DOWNLOAD_FOLDER$/ForecasterNotes</downloadFolderForecasterNotes> <downloadFolderConfiguration>$ARCHIVE_DOWNLOAD_FOLDER$/Configuration</downloadFolderConfiguration> <downloadFolderEventAttachments>$ARCHIVE_DOWNLOAD_FOLDER$/Attachments</downloadFolderEventAttachments> <downloadFolderSnapShots>$ARCHIVE_DOWNLOAD_FOLDER$/SnapShots</downloadFolderSnapShots> <downloadFolderProducts>$ARCHIVE_DOWNLOAD_FOLDER$/Products</downloadFolderProducts> </downloadFolders> <archiveImportWorkflowId>Archive_Import</archiveImportWorkflowId> <hideDownloadDataSetsTab>true</hideDownloadDataSetsTab> <hideCreateEventTab>true</hideCreateEventTab> <hideSearchEventTab>true</hideSearchEventTab> <hideArchiveDatabaseTab>true</hideArchiveDatabaseTab> <hideArchiveUploadTab>true</hideArchiveUploadTab> <archiveUploadIdMapId>id</archiveUploadIdMapId> <initialSearchTimePeriod start="-1000" end="1" unit="day"></initialSearchTimePeriod> <dataSearchProperty> <propertyId>aVeryLongPropertyId</propertyId> <displayName>propertyA</displayName> </dataSearchProperty> <dataSearchProperty> <propertyId>propertyB</propertyId> </dataSearchProperty> </archiveModuleDisplay>
Custom configuration
If the archive is not installed according to the conventions desribed in the installation manual than it might be needed to configure custom endpoint for the archive-webservices which are used by the archive panel. This custom configuration can be done in the tag "archiveServer", below an example is given.
<archiveServer> <archiveWebService>http://dummy_server:8080/deltares-archive-server</archiveWebService> <archiveConfigFileUrl>http://dummy_server:8080/thredds/fileServer/Config</archiveConfigFileUrl> <eventsFileServerUrl>http://dummy_server:8080/thredds/fileServer/Events</eventsFileServerUrl> <dataFileServerUrl>http://dummy_server:8080/thredds/fileServer/archive</dataFileServerUrl> </archiveServer>
Archive Database
A mongo database plugin can be added to the archives to store scalar data. This tab is only available if you added a <archiveDatabase> to your config. To search for or download data from the database, different filters can (or must be used). The list of areas available is loaded from the configuration.
When the user selects both an area and a time series type, the list of sources available will be populated. When a source is selected, the available time steps, parameters and module instance ids will be loaded from the database.
To search for data sets, area, time series type and source need to be selected. You can also filter the results by time steps, parameters and module instance ids, but these fields aren’t mandatory: if nothing is selected, data for all of the available options will be included in the search result. To search for data, klick the magnifying glass on the left side of the panel.
The summary of the available data sets will be displayed in the text field at the bottom of the panel.
You can download the selected data by clicking the download button (under the search button). When the download is complete, the summary will be logged and also displayed in the panel.
When you are downloading data from the search and download datasets panel, if part or all of the time series are stored in the mongo database, it will be automatically downloaded along with the data stored in the open archive.
Config example:
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>url</catalogueUrl> <elasticSearchUrl>url</elasticSearchUrl> <archiveDatabase> <archiveDatabaseUrl>url</archiveDatabaseUrl> <user>user</user> <password>pass</password> <pluginClass>nl.fews.archivedatabase.mongodb.MongoDbArchiveDatabase</pluginClass> <pluginFactoryMethod>create</pluginFactoryMethod> <openDatabasePluginPanelBinDir>d:\mongdb-plugin\</openDatabasePluginPanelBinDir> </archiveDatabase> </archives>
Upload to archive
It is possible to add quality controlled external historical data to you existing archive. This can be done by the following procedure.
1) Import the data you want to add in your stand-alone FEWS application,
2) Verify the data by checking it in the Plots display and edit the data where needed.
3) When the data is verified and approved then you can upload the data to your archive by using the archive display. A new tab (available from 2022.02) Upload to Archive is available. Start the achive display and select the Upload to Archive tab.
4) Select the time series you want to upload by using the data viewer. You can select the period you want to upload by selecting the period in the panel. In addition you should select the area to which you want to add the data.
5) Before you can upload the data you should press the "Check availability in catalog" button. After pressing the button the display will check for which periods the selected time series are already available in the archive. In addition it will check which time series are edited in the periods for which the data is already archived. In the example below there is already data available in the archive for a part of the selected period. The last selected time series also has 2 data edits in the period for which the data is already archived.
6) After step 5 the button "Upload to archive" will enabled. Note that when you change the selected time series the button will be unavailable again and that you should check the availability of the data again. When you press the button "Upload to archive" the selected time series will be uploaded to archive. For the periods for which the time series are not already archived to time series will be exported to netcdf and uploaded to the archive. For the periods for which the data is already archived only the edited values will be uploaded in pi-xml format to the archive. Both uploads are handled by the webservices of the archive. To insert the edited data into the archive a workflow in FEWS must be executed. To insert the newly added netcdf-files into the archive a task at the archive server must be executed.
