snapshot-based peer recovery warning despite not using indices.recovery.use_snapshots=true #86705
Description
Elasticsearch Version
Version: 8.2.0, Build: default/tar/b174af62e8dd9f4ac4d25875e9381ffe2b9282c5/2022-04-20T10:35:10.180408517Z, JVM: 18
Installed Plugins
No response
Java Version
bundled
OS Version
MacOS Darwin Kernel Version 21.4.0
Problem Description
Elasticsearch throws snapshot-based peer recovery warnings despite setting "indices.recovery.use_snapshots": false + having "use_for_peer_recovery": false set on the repository.
Steps to Reproduce
This could be tested with a 2x node cluster consisting of 1x master/data and 1x data node.
Create a snapshot repository with "use_for_peer_recovery": false:
PUT _snapshot/my_snapshot_repo { "type": "fs", "settings": { "location": "backups", "use_for_peer_recovery": false } } Ensure "indices.recovery.use_snapshots": false is set:
PUT _cluster/settings { "persistent": { "indices.recovery.use_snapshots": false } } Stop the data-only node and remove the contents of its ./data/indices directory, then restart the node.
The node should log warnings about not being able to perform snapshot-based peer recoveries.
A couple other observations:
- The log message (below) is a bit confusing because
indices.recovery.max_concurrent_snapshot_file_downloadshas been left at defaults. - These warnings can also be seen when using
"indices.recovery.use_snapshots": true+"use_for_peer_recovery": truebut no valid snapshot of the index exists.
Logs (if relevant)
[2022-05-11T14:51:14,538][INFO ][o.e.n.Node ] [node-c] started [2022-05-11T14:51:15,834][WARN ][o.e.i.r.RecoverySettings ] [node-c] Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25] [2022-05-11T14:51:24,499][WARN ][o.e.i.r.RecoverySettings ] [node-c] Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25] [2022-05-11T14:51:25,005][WARN ][o.e.i.r.RecoverySettings ] [node-c] Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25] [2022-05-11T14:51:32,363][WARN ][o.e.i.r.RecoverySettings ] [node-c] Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25] [2022-05-11T14:51:48,103][WARN ][o.e.i.r.RecoverySettings ] [node-c] Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]