ElasticSearch cluster master data deleted

Question

I have an Elastic cluster running on Kubernetes with 3 master nodes (2 with persistent storage and 1 voting-only node without persistent storage) and 20 data nodes. I accidentally deleted the data on one master node with persistent storage, but I successfully rejoined the master node back to the cluster using the same name, though with a new UUID that was generated. The cluster has been working fine for a few weeks.

However, my question is: will it continue to work after a complete stop and start of the cluster? Or will it require the original master UUID from the initial bootstrap?

Thanks.

Answer 1

I haven't been in this situation, but AFAIK it should be fine to restart the cluster. As a final confirmation, I would check the cluster health and state.

GET /_cluster/health GET /_cluster/state 

If the cluster health is stable and the old UUID has been removed from the state, restarting the entire cluster should work without issues. The cluster will re-elect the master and recognize the current nodes.

With default settings for discovery.zen.fd.ping_interval (1s), discovery.zen.fd.ping_timeout (30s) & discovery.zen.fd.ping_retries (3) it should take less than two minutes for the master node to be marked as failed, and after a while (within minutes) completely removed from the state.

You said you did this weeks ago. The old UUID should be long gone, but depends on the Zen discovery configuration. Therefore, it is better to confirm the cluster state before the restart.