Basic configuration

The Archives.xml file, located in the SystemConfigFiles folder, contains the configuration of the connection between your SA, OC or FSS and the Open Archive. 

Catalogue

For SA and OC, the configuration consists of only the catalogueUrl (assuming the archive is configured according to the standard):

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>http://dummy_server:8080/deltares-archive-server/catalogue</catalogueUrl>
</archives>

Seamless integration

Data from the Open Archive can be accessed through Seamless Integration. This means that when data is not found in the Central Database, Delft-FEWS will search for this data in the Open Archive instead.

The Seamless Integration functionality is available for:

• external historical scalar time series only , both scalar and gridded
• in the Time Series Display and Grid Display, and can be activated here with the button “connect to external data source”
  • In the TimeSeriesDisplay, the data within the view period of the graph is downloaded and displayed as it becomes available
  • In the GridDisplay, all the meta data is download and displayed, the data itself is only downloaded for the time step in the slider. 
    • E.g. the colors on the time slider are immediately updated.
    • The data itself is downloaded only for a time step selected in the slider (to avoid unnecessary large data transfers).

To enable seamless integration, the location of Thredds needs to be configured, next to the location of the Catalogue:

• catalogueUrl: The internal elastic search catalogue is used to find archived data for seamless integration (not the Geonetwork catalogue). 
• fileServerUrl: The fileserver THREDDS used to download the data from the archive for seamless integration.
• dataFolder (optional): This location is used by the pi-service (rest and soap) to read the data.
  • If configured the data is read from the folder directly and not from THREDDS.
  • Reading the data directly from disk is faster than reading the data from THREDDS.
  • The datafolder should be the root of the archive datafolder, as it is on the server on which the pi-service is configured.

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl>
	<fileServerUrl>http://dummy_server:8081/thredds/fileServer/data</fileServerUrl>
    <dataFolder>d:\fews\archive\data\</dataFolder>
</archives>

Test of the Archive url configuration

catalogueUrl

To test the catalogueUrl, paste the url in an internet browser. 

• If the config is correct the response below will be visible (or something similar).
  The response either indicates that the catalogue is moved to another URL (releases prior to august 2020) or the response will look like the below.

This url was used to host the embedded catalogue prior to the 201802 release.
Now it is used to verify that the correct catalogueURL is configured correctly in the Archives.xml
The URL http://localhost:7080/deltares-archive-server/catalogue should be used as the catalogueURL in the Archives.xml

fileServerUrl

To test the fileServerUrl append /catalog.html  to the url and paste in an Internet browser. 

• If the config is correct a list of the files and directories in the root of the data folder in the archive will be visible. 

elasticSearchUrl

You can test the elasticSearchUrl (optional) by pasting the URL in a browser. 

• If the config is correct the response below will be visible (or something similar).

Optional configuration

elasticSearchUrl - direct access

By default Delft-FEWS will assume that the elastic catalogue is running at the same server as the archive web application at the port 9200.
This can be changed with the configuration option elasticSearchUrl. 

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl>
	<elasticSearchUrl>http://dummy_server:9800</elasticSearchUrl>
</archives>

elasticSearchUrl - reverse proxy

A Delft-FEWS OC or SA needs access to the catalogue to search for data in the archive, which requires direct access to elastic, introducing an attack vector.
To avoid this Deltares Open Archive contains a read-only reverse proxy inside the Archive web application (since August 2020), which:

• redirects only the limited set of read-only elastic requests which an FEWS OC or SA requires. 
• is available at the relative URL /elastic in the Archive web application.

To connect the OCs through the reverse proxy instead of directly to elastic, configure an elasticSearchUrl: 

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl>
	<elasticSearchUrl>http://dummy_server:7080/deltares-archive-server/elastic</elasticSearchUrl>
</archives>

enabled

The Enabled element (optional) can control which components have direct access to the Archive, see the example below:

• A global property $ARCHIVE_ENABLED$ is used to control if the archive connection is enabled.
• This allows for example OC clients or FSS nodes (each having their own global properties) to not allow connections to the archive in specific cases.
• Setting this option to FALSE, will of course disable seamless integration on these components.

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<enabled>$ARCHIVE_ENABLED$</enabled>    
	<catalogueUrl>http://dummy_server:7080/deltares-archive-server/catalogue</catalogueUrl>
	<elasticSearchUrl>http://dummy_server:7080/deltares-archive-server/elastic</elasticSearchUrl>
</archives>

archiveDatabase

It is possible to store scalar data inside a database instead of in netcdf files. 

• The details of the database (URL, username and password) and the details about how to load the plugin for the archive database should be configured in Archives.xml.
• At the moment there is a plugin available for MongoDB. 

  • The archiveDatabaseUrl and the username and password are dependent on the system setup.

  • If you use the MongoDB-plugin then you should use the pluginClass and pluginFactoryMethod as shown in the example.

  • The plugin can be provided by Deltares on request.

A configuration example for the MongoDB plugin is available below:

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>https://dummy_server/deltares-archive-server/</catalogueUrl>
	<elasticSearchUrl>https://dummy_server/deltares-archive-server/elastic</elasticSearchUrl>
	<archiveDatabase>
		<archiveDatabaseUrl>mongodb://server:port/fewsarchive?authSource=admin</archiveDatabaseUrl>
		<user>myuser</user>
		<password>mypass</password>
		<pluginClass>nl.fews.archivedatabase.mongodb.MongoDbArchiveDatabase</pluginClass>
		<pluginFactoryMethod>create</pluginFactoryMethod>
	    <openDatabasePluginPanelBinDir>d:\mongdb-plugin\</openDatabasePluginPanelBinDir>
	</archiveDatabase>
</archives>

pluginDir

You can configure a custom directory to install the jar-files of the plugin in. If you do not configure a custom pluginDir then you should located the plugin .jar file in the same directory as the FEWS jars.

uploadEditedDataToOpenArchive - Uploading edited data to the archive

When data is edited in Delft-FEWS, the data edits are not send to the archive (default behavior).

• This default behavior can be chanced with uploadEditedDataToOpenArchive.

• The location attribute function areaLocationAttributeFunction (required when using this feature)is used to determine to which area (one or more) each location belongs.
• In addition a workflow needs to scheduled to insert the data edits into the archive.
  More details can be found here 22-4 Merging edited data - DELFT-FEWS Documentation - Deltares Public Wiki

Below a configuration example of Archives.xml

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>http://localhost:7080/deltares-archive-server/catalogue</catalogueUrl>
	<elasticSearchUrl>http://localhost:7080/deltares-archive-server/elastic</elasticSearchUrl>
	<areaLocationAttributeFunction>@station:REPORT_FOLDER@</areaLocationAttributeFunction>
	<uploadEditedDataToOpenArchive>true</uploadEditedDataToOpenArchive>
</archives>   

customExportFolders

Data is exported to the open archive according to fixed folder structure (see 22-2 Export to Deltares Open Archive). 
It is possible to configure a custom export folder for each data type in the archive (since 2023.02).
You can use plain text in the folder structure, or use one or more of the following (see also the example below).

date/time pattern

You can use date/time pattern in the folders structure.

• The pattern will be parsed during export by using the time relevant for the export.
• The time that is used to parse the date/time pattern is dependent on the type of export.
• Below an overview of the time used for each export type.  

tags

A set of tags like %areaId% are supported and will be parsed during the export.
Not all tags are supported for each type of export, see table below.

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<catalogueUrl>https://fews-ut-ar-202301.avi.deltares.nl/deltares-archive-server/catalogue</catalogueUrl>
	<elasticSearchUrl>https://fews-ut-ar-202301.avi.deltares.nl/deltares-archive-server/elastic</elasticSearchUrl>
	<dataFolder>C:\fews\archive\data\</dataFolder>
	<customExportFolders>
		<products>yyyy/MM'/%areaId%/%sourceId%/'dd'/products/%productId%_%currentTime%'</products>
		<config>yyyy/MM'/%areaId%/%revisionId%'</config>
		<forecasterNotes>yyyy/MM'/%areaId%/test/'dd'/forecasterNotes'</forecasterNotes>
		<logMessages>yyyy/MM/dd'%areaId%/logmessages'</logMessages>
		<ratingCurves>yyyy/MM/dd'%areaId%/ratingcurves'</ratingCurves>
		<snapShot>yyyy/MM/dd'%areaId%/snapshot'</snapShot>
		<externalHistorical>yyyy/MM'/%areaId%/%sourceId%/'dd'/observed_%valueType%'</externalHistorical>
		<externalForecast>yyyy/MM'/%areaId%/%sourceId%/'dd'/externalForecast/%sourceId%_'HHmmss</externalForecast>
		<simulatedForecast>yyyy'/%taskRunId%/%areaId%/%sourceId%/%whatIfId%/%workflowId%_%forecastTime%_%currentTime%'</simulatedForecast>
		<simulatedHistorical>yyyy/MM'/%areaId%/%sourceId%/'dd'/simulatedHistorical_%valueType%'</simulatedHistorical>
	</customExportFolders>
</archives>