Catalogue is in read-only mode
The latest version of the archive admin console indicates if your catalogue is in read-only mode.
If you run the harvester when the catalogue is in read-only mode then you will see the following errors in the log-files of the archive.
[0]: index [metadatafiles], type [search], id [ovl0nH8BF4WRu26gnjcI], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[1]: index [simulatedmetadata_seamless], type [seamless_integration_search], id [o_l0nH8BF4WRu26gnjcI], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
If the catalogue is in read-only mode then the catalogue can no longer be updated. Elastic will switch to read-only mode when the available disk-space is less then 5%. Note that elastic will stay in read-only mode even when you make more disk space available.
First thing to do when you see these types of errors is to configure elastic so that it doesn't switch quickly to read-only mode.
This can be done by manually configuring the thresholds in the elasticsearch.yml. This file can be found in the config folder of elastic.
cluster.routing.allocation.disk.watermark.low: "20gb"
cluster.routing.allocation.disk.watermark.high: "15gb"
cluster.routing.allocation.disk.watermark.flood_stage: "10gb"
Note that the mentioned thresholds above should be configured in such a way that cluster.routing.allocation.disk.watermark.low is higher than cluster.routing.allocation.disk.watermark.high and that cluster.routing.allocation.disk.watermark.high should be higher than cluster.routing.allocation.disk.watermark.flood_stage.
The config above will ensure that elastic only switches to read-only mode when there is less then 10gb available.
To force elastic to leave the read-only mode when there is more disk space available or when the settings in the elasticsearch.yml are adjusted you can execute the following commands at the elastic server.
windows
curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_all/_settings -d "{ \"index.blocks.read_only_allow_delete\": null }"
linux
curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_all/_settings -d '{ "index.blocks.read_only_allow_delete": null }'
Another option is to delete the current catalogue and rebuild it from scratch. To do this you should stop the tomcat instance which hosts the archive.
When the archive is stopped you should remove the data folder in the elasticsearch installation directory.
After that you can the archive again.
To rebuild the catalogue you should go to Archive tasks tab and start the harvester.
Improve harvester speed
The harvesting process can be quite consuming. For simulated data and external forecasts both the available netcdf-files and the available time series are stored in the catalogue. The indexing of the time series of simulated data and external forecasts consumes a lot of time.
These indeces are only needed when you want to retrieve simulated data or external forecasts from the archive by using the pi webservice. This feature is only used by limited amount of clients. If you dont use this feature then you can disable this indexing by the following configuration option the archiveServerConfig.xml. Below an example.
<archiveServer xmlns:arc="http://www.wldelft.nl/fews/archive" xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/archiveServer.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<archiveEnvironmentName>$prop1$</archiveEnvironmentName>
<baseUrlFileServer>http://localhost:8080/thredds/fileServer/Archive</baseUrlFileServer>
<baseUrlArchiveRootFolder>file:///d:\fews\archive\</baseUrlArchiveRootFolder>
<indexForecastTimeSeries>false</indexForecastTimeSeries>
</archiveServer>
