Catalogue read-only mode
| Warning |
|---|
If the catalogue is in read-only mode |
...
then the catalogue can no longer be updated. Elastic will switch to read-only mode when the available disk-space is less then 5%. Note that elastic will stay in read-only mode |
...
even when you make more disk space available. |
If you run the harvester when the catalogue is in read-only mode then you will see the following errors in the log-files of the archive.
[0]: index [metadatafiles], type [search], id [ovl0nH8BF4WRu26gnjcI], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[1]: index [simulatedmetadata_seamless], type [seamless_integration_search], id [o_l0nH8BF4WRu26gnjcI], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
If the The latest version of the archive admin console indicates if your catalogue is in read-only mode then the catalogue can no longer be updated. Elastic will switch to .
Control when read-only mode is activated
Elastic can be configured when to when the available disk-space is less then 5%. Note that elastic will stay in read-only mode even when you make more disk space available.First thing to do when you see these types of errors is to configure elastic so that it doesn't switch quickly to read-only mode. This can be done by manually configuring the thresholds in the elasticsearch.yml. This file can be found in the config folder of elastic.
| Code Block | ||
|---|---|---|
| ||
cluster.routing.allocation.disk.watermark.low: "20gb" |
...
cluster.routing.allocation.disk.watermark.high: "15gb" |
...
cluster.routing.allocation.disk.watermark.flood_stage: "10gb" |
Note that the mentioned thresholds above should be configured in such a way that NB. cluster.routing.allocation.disk.watermark.low is must be higher than cluster.routing.allocation.disk.watermark.high and that cluster.routing.allocation.disk.watermark.high should be higher than cluster cluster.routing.allocation.disk.watermark.flood_stage.
The config above will ensure that elastic only switches to read-only mode when there is less then 10gb available. To force elastic to leave the read-only mode when there is more disk space available or when the settings in the elasticsearch.yml are adjusted you can execute the following commands at the elastic server.windows
linux
| Code Block | ||
|---|---|---|
| ||
curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_all/_settings -d |
...
'{ |
...
"index.blocks.read_only_allow_delete |
...
": null |
...
}' |
Windows
| Code Block | ||
|---|---|---|
|
...
curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_all/_settings -d |
...
"{ \"index.blocks.read_only_allow_delete\": null |
...
}" |
Rebuild catalogue from scratch
Another option is to delete the current catalogue and rebuild it from scratch. To do this you should
- stop the tomcat instance which hosts the archive.
...
- remove the data folder in the elasticsearch installation directory.
...
- start the archive again.
...
- start the harvester from Archive tasks tab
...
- to rebuild the
...
- catalogue.
...
Speed up harvester
...
The harvesting process can be quite time consuming. For simulated data and external forecasts both the available netcdfnetCDF-files and the available time series are stored in the catalogue. The indexing of the time series of simulated data and external forecasts consumes a lot of time. These indeces indexes are only needed when you want to retrieve simulated data or external forecasts from the archive by using the pi webserviceDelft-FEWS web services. This feature is only used by limited amount of clients. If you dont do not use this feature then you can disable this indexing by the following configuration option the archiveServerConfig.xml. Below an example.
| Code Block | ||
|---|---|---|
| ||
<archiveServer xmlns:arc="http://www.wldelft.nl/fews/archive" xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/archiveServer.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> |
...
<archiveEnvironmentName>$prop1$</archiveEnvironmentName> |
...
...
<baseUrlFileServer>http://localhost:8080/thredds/fileServer/Archive</ |
...
baseUrlFileServer> <baseUrlArchiveRootFolder>file:///d:\fews\archive\</baseUrlArchiveRootFolder> |
...
<indexForecastTimeSeries>false</indexForecastTimeSeries> |
...
</archiveServer> |