Basic configuration
The Archives.xml file, located in the SystemConfigFiles folder, contains the configuration of the connection between your SA, OC or FSS and the Open Archive.
Catalogue
For SA and OC, the configuration consists of only the catalogueUrl (assuming the archive is configured according to the standard):
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>http://dummy_server:8080/deltares-archive-server/catalogue</catalogueUrl> </archives>
A common mistake is that the catalogueURL used does not end with catalogue!
Seamless integration
Data from the Open Archive can be accessed through Seamless Integration. This means that when data is not found in the Central Database, Delft-FEWS will search for this data in the Open Archive instead.
The Seamless Integration functionality is available for:
- external historical scalar time series only , both scalar and gridded
- in the Time Series Display and Grid Display, and can be activated here with the button “connect to external data source”
- In the TimeSeriesDisplay, the data within the view period of the graph is downloaded and displayed as it becomes available
- In the GridDisplay, all the meta data is download and displayed, the data itself is only downloaded for the time step in the slider.
- E.g. the colors on the time slider are immediately updated.
- The data itself is downloaded only for a time step selected in the slider (to avoid unnecessary large data transfers).
The Delft-FEWS pi-webservices contain additional functionality for external historical and simulated time series (see FEWS PI REST Web Service#GETtimeseries)
To enable seamless integration, the location of Thredds needs to be configured, next to the location of the Catalogue:
- catalogueUrl: The internal elastic search catalogue is used to find archived data for seamless integration (not the Geonetwork catalogue).
- fileServerUrl: The fileserver THREDDS used to download the data from the archive for seamless integration.
- dataFolder (optional): This location is used by the pi-service (rest and soap) to read the data.
- If configured the data is read from the folder directly and not from THREDDS.
- Reading the data directly from disk is faster than reading the data from THREDDS.
- The datafolder should be the root of the archive datafolder, as it is on the server on which the pi-service is configured.
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl> <fileServerUrl>http://dummy_server:8081/thredds/fileServer/data</fileServerUrl> <dataFolder>d:\fews\archive\data\</dataFolder> </archives>
Test of the Archive url configuration
catalogueUrl
To test the catalogueUrl
, paste the url in an internet browser.
- If the config is correct the response below will be visible (or something similar).
The response either indicates that the catalogue is moved to another URL (releases prior to august 2020) or the response will look like the below.
This url was used to host the embedded catalogue prior to the 201802 release. Now it is used to verify that the correct catalogueURL is configured correctly in the Archives.xml The URL http://localhost:7080/deltares-archive-server/catalogue should be used as the catalogueURL in the Archives.xml
fileServerUrl
To test the fileServerUrl
append /catalog.html
to the url and paste in an Internet browser.
- If the config is correct a list of the files and directories in the root of the data folder in the archive will be visible.
elasticSearchUrl
You can test the elasticSearchUrl (optional) by pasting the URL in a browser.
- If the config is correct the response below will be visible (or something similar).
Optional configuration
elasticSearchUrl - direct access
By default Delft-FEWS will assume that the elastic catalogue is running at the same server as the archive web application at the port 9200.
This can be changed with the configuration option elasticSearchUrl
.
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl> <elasticSearchUrl>http://dummy_server:9800</elasticSearchUrl> </archives>
elasticSearchUrl - reverse proxy
A Delft-FEWS OC or SA needs access to the catalogue to search for data in the archive, which requires direct access to elastic, introducing an attack vector.
To avoid this Deltares Open Archive contains a read-only reverse proxy inside the Archive web application (since August 2020), which:
- redirects only the limited set of read-only elastic requests which an FEWS OC or SA requires.
- is available at the relative URL /elastic in the Archive web application.
To connect the OCs through the reverse proxy instead of directly to elastic, configure an elasticSearchUrl:
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl> <elasticSearchUrl>http://dummy_server:7080/deltares-archive-server/elastic</elasticSearchUrl> </archives>
Both the catalogueUrl and the elasticSearchUrl tags should be configured!
enabled
The Enabled element (optional) can control which components have direct access to the Archive, see the example below:
- A global property $ARCHIVE_ENABLED$ is used to control if the archive connection is enabled.
- This allows for example OC clients or FSS nodes (each having their own global properties) to not allow connections to the archive in specific cases.
- Setting this option to FALSE, will of course disable seamless integration on these components.
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd"> <enabled>$ARCHIVE_ENABLED$</enabled> <catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl> <elasticSearchUrl>http://dummy_server:7080/deltares-archive-server/elastic</elasticSearchUrl> </archives>
archiveDatabase
It is possible to store scalar data inside a database instead of in netcdf files.
- The details of the database (URL, username and password) and the details about how to load the plugin for the archive database should be configured in Archives.xml.
- At the moment there is a plugin available for MongoDB.
The archiveDatabaseUrl and the username and password are dependent on the system setup.
If you use the MongoDB-plugin then you should use the pluginClass and pluginFactoryMethod as shown in the example.
The plugin can be provided by Deltares on request.
A configuration example for the MongoDB plugin is available below:
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>https://dummy_server/deltares-archive-server/</catalogueUrl> <elasticSearchUrl>https://dummy_server/deltares-archive-server/elastic</elasticSearchUrl> <archiveDatabase> <archiveDatabaseUrl>mongodb://server:port/fewsarchive?authSource=admin</archiveDatabaseUrl> <user>myuser</user> <password>mypass</password> <pluginClass>nl.fews.archivedatabase.mongodb.MongoDbArchiveDatabase</pluginClass> <pluginFactoryMethod>create</pluginFactoryMethod> <openDatabasePluginPanelBinDir>d:\mongdb-plugin\</openDatabasePluginPanelBinDir> </archiveDatabase> </archives>
pluginDir
You can configure a custom directory to install the jar-files of the plugin in. If you do not configure a custom pluginDir then you should located the plugin .jar file in the same directory as the FEWS jars.
uploadEditedDataToOpenArchive - Uploading edited data to the archive
When data is edited in Delft-FEWS, the data edits are not send to the archive (default behavior).
- This default behavior can be chanced with
uploadEditedDataToOpenArchive.
- The location attribute function
areaLocationAttributeFunction
(required when using this feature)
is used to determine to which area (one or more) each location belongs. - In addition a workflow needs to scheduled to insert the data edits into the archive.
More details can be found here 22-4 Merging edited data - DELFT-FEWS Documentation - Deltares Public Wiki
Below a configuration example of Archives.xml
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>http://localhost:7080/deltares-archive-server/catalogue</catalogueUrl> <elasticSearchUrl>http://localhost:7080/deltares-archive-server/elastic</elasticSearchUrl> <areaLocationAttributeFunction>@station:REPORT_FOLDER@</areaLocationAttributeFunction> <uploadEditedDataToOpenArchive>true</uploadEditedDataToOpenArchive> </archives>
customExportFolders
Data is exported to the open archive according to fixed folder structure (see 22-2 Export to Deltares Open Archive).
It is possible to configure a custom export folder for each data type in the archive (since 2023.02).
You can use plain text in the folder structure, or use one or more of the following (see also the example below).
date/time pattern
You can use date/time pattern in the folders structure.
- The pattern will be parsed during export by using the time relevant for the export.
- The time that is used to parse the date/time pattern is dependent on the type of export.
- Below an overview of the time used for each export type.
Time used | |
External historical | start time exported data |
External forecast | external forecast time |
Simulated historical | start time exported data |
Simulated forecast | time zero |
Products | time zero |
Config | creation time latest revision |
Rating curves | current time |
Snapshot | current time |
Log messages | start time exported data |
Forecaster notes | start time exported data |
tags
A set of tags like %areaId% are supported and will be parsed during the export.
Not all tags are supported for each type of export, see table below.
areaId | sourceId | currentTime | taskRunId | whatIfId | workflowId | supplied attributes | revisionId | |
External historical | x | x | ||||||
External forecast | x | x | ||||||
Simulated historical | x | x | ||||||
Simulated Forecast | x | x | x | x | x | x | ||
Products | x | x | x | x | ||||
Config | x | x | ||||||
Rating curves | x | |||||||
Snapshot | x | |||||||
Log messages | x | |||||||
Forecaster notes | x |
<?xml version="1.0" encoding="UTF-8"?> <archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd"> <catalogueUrl>https://fews-ut-ar-202301.avi.deltares.nl/deltares-archive-server/catalogue</catalogueUrl> <elasticSearchUrl>https://fews-ut-ar-202301.avi.deltares.nl/deltares-archive-server/elastic</elasticSearchUrl> <dataFolder>C:\fews\archive\data\</dataFolder> <customExportFolders> <products>yyyy/MM'/%areaId%/%sourceId%/'dd'/products/%productId%_%currentTime%'</products> <config>yyyy/MM'/%areaId%/%revisionId%'</config> <forecasterNotes>yyyy/MM'/%areaId%/test/'dd'/forecasterNotes'</forecasterNotes> <logMessages>yyyy/MM/dd'%areaId%/logmessages'</logMessages> <ratingCurves>yyyy/MM/dd'%areaId%/ratingcurves'</ratingCurves> <snapShot>yyyy/MM/dd'%areaId%/snapshot'</snapShot> <externalHistorical>yyyy/MM'/%areaId%/%sourceId%/'dd'/observed_%valueType%'</externalHistorical> <externalForecast>yyyy/MM'/%areaId%/%sourceId%/'dd'/externalForecast/%sourceId%_'HHmmss</externalForecast> <simulatedForecast>yyyy'/%taskRunId%/%areaId%/%sourceId%/%whatIfId%/%workflowId%_%forecastTime%_%currentTime%'</simulatedForecast> <simulatedHistorical>yyyy/MM'/%areaId%/%sourceId%/'dd'/simulatedHistorical_%valueType%'</simulatedHistorical> </customExportFolders> </archives>