Disk-based shard allocation

Topics

• What is a watermark

• Configuring watermark stages

• Flood stage and read-only mode

Since version 7.4.3 of Logserver a feature was added to keep you informed about your disk usage. Disk-based shard allocation settings are used to determine the GUI behaviour.

These labels will appear when a certain watermark level is exceeded. Its function is to inform you to take action. Maybe tweak your retention settings or add disk space to your servers.

With the “FLOOD STAGE”, non-administrative users will lose the ability to log into GUI, and those already logged in will be redirected to the status page. An administrator will still be able to browse and will have a chance to take action to fix the situation.

This is because with “FLOOD STAGE”, the system switches into a read-only mode. No new documents can be created or updated.

What is a watermark

1. Low stage

   Controls the low watermark for disk usage. When exceeded Logserver will not allocate shards to that node. This setting does not affect the primary shards of newly-created indices but will prevent their replicas from being allocated.

2. High stage

   Controls the high watermark. When exceeded Logserver will attempt to relocate shards away from a node whose disk exceeds it. This setting affects the allocation of all shards, whether previously allocated or not.

3. Flood stage

   This setting is a last resort to prevent nodes from running out of disk space - which should be at all costs avoided.

   Controls the flood stage watermark. Logserver enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. The index block is automatically released when the disk utilization falls below the high watermark.

Even if you are running Logserver in a single node environment, configuring the above is highly recommended.

While the GUI will always inform you about your disk usage, to take advantage of the above benefits you will need to configure the thresholds yourself especially if you are updating from an earlier Logserver version.

Configuring watermark stages

1. On fresh 7.4.3 or newer installations of Logserver these are the default watermark values:

   • low: 95%

   • high: 97%

   • flood: 99%

2. To change them, you can use Logserver API like in the following example:

   curl localhost:9200/_cluster/settings -XPUT -H 'content-type: application/json' -d '{
     "persistent": {
       "cluster.routing.allocation.disk.threshold_enabled": "true",
       "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
       "cluster.routing.allocation.disk.watermark.high": "25gb",
       "cluster.routing.allocation.disk.watermark.low": "50gb"
     }
   }'
   

   With that the “low stage” starts when the disk space for Logserver indices falls below the 50 GB mark; “high” below 25 GB and “flood” below 10 GB.

3. You can also use percentages for that but not both - you cannot mix byte values and percentages.

4. You can also modify Logserver configuration file /etc/logserver/logserver.yml:

   cluster.routing.allocation.disk.threshold_enabled: true
   cluster.routing.allocation.disk.watermark.low: "50gb"
   cluster.routing.allocation.disk.watermark.high: "25gb"
   cluster.routing.allocation.disk.watermark.flood_stage: "10gb"
   

   Take in mind you will have to change those values on all of your cluster nodes this way. Restart is not required as API was used earlier.

   Important! Thanks to that, settings will be protected against a state failure.

Flood stage and read-only mode

When the cluster reaches the “flood stage”, it is automatically put into a read-only mode. It prevents a cluster failure - reaching 100% disk usage on a Logserver cluster is a critical issue!

The best way to remove the read-only mode is to make more disk space available for the cluster - either by adding more space or removing data (indices) from it. When the cluster reaches the “low stage” or below, it will automatically switch from the read-only mode.

Adjusting the levels with the above API examples to suit your installation is another option.

Disabling the threshold checking is not recommended! Also, changing the “threshold_enabled” setting to “false” does not turn the read-only mode off automatically!