[FORBIDDEN/12/index read-only / allow delete errors
When you see these types of errors in your log-files then elastic has switched to read-only mode. This will cause that the catalogue can no longer be updated. Elastic will switch to read-only mode when the available disk-space is less then 5%. Note that elastic will stay in read-only mode even when you make more disk space available.
First thing to do when you see these types of errors is to configure elastic so that it doesn't switch quickly to read-only mode.
This can be done by manually configuring the thresholds in the elasticsearch.yml. This file can be found the config folder of elastic.
cluster.routing.allocation.disk.watermark.low: "20gb"
cluster.routing.allocation.disk.watermark.high: "15gb"
cluster.routing.allocation.disk.watermark.flood_stage: "10gb"
Note that the mentioned thresholds above should be configured in such a way that cluster.routing.allocation.disk.watermark.low is higher than cluster.routing.allocation.disk.watermark.high and that cluster.routing.allocation.disk.watermark.high should be higher than cluster.routing.allocation.disk.watermark.flood_stage.
The config above will ensure that elastic only switches to read-only mode when there is less then 10gb available.
To force elastic to leave the read-only mode when there is more disk space available or when the settings in the elasticsearch.yml are adjusted you can execute the following commands at the elastic server.
windows
curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_all/_settings -d "{ \"index.blocks.read_only_allow_delete\": null }"
linux
curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_all/_settings -d '{ "index.blocks.read_only_allow_delete": null }'
Another option is to delete the current catalogue and rebuild it from scratch. To do this you should stop elastic.
This can be done in the tab catalogue in the archive admin console
When the catalogue is stopped you should remove the data folder in the elasticsearch installation directory. After that you can elasticsearch again.
To rebuild the catalogue you should go to Archive tasks tab and start the harvester task.
Improve harvester speed
The harvesting process can be quite consuming. For simulated data and external forecasts both the available netcdf-files and the available time series are stored in the catalogue. The indexing of the time series of simulated data and external forecasts consumes a lot of time.
These indeces are only needed when you want to retrieve simulated data or external forecasts from the archive by using the pi webservice. This feature is only used by limited amount of clients. If you dont use this feature then you can disable this indexing by the following configuration option the archiveServerConfig.xml. Below an example.
<archiveServer xmlns:arc="http://www.wldelft.nl/fews/archive" xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/archiveServer.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<archiveEnvironmentName>$prop1$</archiveEnvironmentName>
<baseUrlFileServer>http://localhost:8080/thredds/fileServer/Archive</baseUrlFileServer>
<baseUrlArchiveRootFolder>file:///d:\fews\archive\</baseUrlArchiveRootFolder>
<indexForecastTimeSeries>false</indexForecastTimeSeries>
</archiveServer>
