Ensure green step in synonyms rule yaml test

[kingherc]

Fixes test issue serverless 2922.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[benwtrent]

Thank you!

[carlosdelest]

Thanks for fixing this @kingherc !

For my education - why fast refresh now requires to wait for green, in this specific case? 🤔

[kingherc]

@carlosdelest with recent changes, searches (I assume including getting synonym rules) will run on search/replica shards in stateless also for fast refresh indices. So we need to wait for green == search shards allocated.

Interestingly the bwc CI on elasticsearch did not pass on the yml now. Investigating the cluster logs, I see it has unassigned shards due to:

[2024-10-09T10:49:24,955][WARN ][o.e.i.c.IndicesClusterStateService] [v8.16.0-3] [test_index][0] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [test_index][0]: Recovery failed on {v8.16.0-3}{fYAWdGI_RLypAvD-ll3rDQ}{nY0Sx-SrRm6EDAq8FdbWHA}{v8.16.0-3}{127.0.0.1}{127.0.0.1:40471}{cdfhilmrstw}{8.16.0}{7000099-8516000}{testattr=test, ml.allocated_processors_double=32.0, ml.machine_memory=126604152832, ml.config_version=12.0.0, ml.max_jvm_size=536870912, ml.allocated_processors=32, xpack.installed=true, transform.config_version=10.0.0} at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$36(IndexShard.java:3366) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$9(StoreRecovery.java:400) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$12(StoreRecovery.java:504) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:188) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:193) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:346) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:399) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:311) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:210) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:175) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:500) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:96) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2498) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway ... 24 more Caused by: java.lang.IllegalArgumentException: failed to build synonyms from [[set1] synonyms_set in .synonyms index] at org.elasticsearch.analysis.common.SynonymTokenFilterFactory.buildSynonyms(SynonymTokenFilterFactory.java:254) ~[?:?] at org.elasticsearch.analysis.common.SynonymTokenFilterFactory.getChainAwareTokenFilterFactory(SynonymTokenFilterFactory.java:197) ~[?:?] at org.elasticsearch.index.analysis.AnalyzerComponents.createComponents(AnalyzerComponents.java:83) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.analysis.ReloadableCustomAnalyzer.reload(ReloadableCustomAnalyzer.java:141) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.analysis.IndexAnalyzers$1.reload(IndexAnalyzers.java:162) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.mapper.MapperService.reloadSearchAnalyzers(MapperService.java:806) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.indices.IndicesService$3.beforeIndexShardRecovery(IndicesService.java:634) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.lambda$iterateBeforeIndexShardRecovery$12(CompositeIndexEventListener.java:303) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:269) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:300) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:327) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1784) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:133) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:421) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] ... 8 more Caused by: java.text.ParseException: Invalid synonym rule at line 1 at org.apache.lucene.analysis.synonym.SolrSynonymParser.parse(SolrSynonymParser.java:74) ~[lucene-analysis-common-9.12.0.jar:?] at org.elasticsearch.analysis.common.SynonymTokenFilterFactory.buildSynonyms(SynonymTokenFilterFactory.java:250) ~[?:?] at org.elasticsearch.analysis.common.SynonymTokenFilterFactory.getChainAwareTokenFilterFactory(SynonymTokenFilterFactory.java:197) ~[?:?] at org.elasticsearch.index.analysis.AnalyzerComponents.createComponents(AnalyzerComponents.java:83) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.analysis.ReloadableCustomAnalyzer.reload(ReloadableCustomAnalyzer.java:141) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.analysis.IndexAnalyzers$1.reload(IndexAnalyzers.java:162) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.mapper.MapperService.reloadSearchAnalyzers(MapperService.java:806) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.indices.IndicesService$3.beforeIndexShardRecovery(IndicesService.java:634) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.lambda$iterateBeforeIndexShardRecovery$12(CompositeIndexEventListener.java:303) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:269) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:300) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:327) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1784) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:133) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:421) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] ... 8 more Caused by: java.lang.IllegalArgumentException: term: baz was completely eliminated by analyzer at org.apache.lucene.analysis.synonym.SynonymMap$Parser.analyze(SynonymMap.java:357) ~[lucene-analysis-common-9.12.0.jar:?] at org.elasticsearch.analysis.common.ESSolrSynonymParser.analyze(ESSolrSynonymParser.java:47) ~[?:?] at org.apache.lucene.analysis.synonym.SolrSynonymParser.addInternal(SolrSynonymParser.java:105) ~[lucene-analysis-common-9.12.0.jar:?] at org.apache.lucene.analysis.synonym.SolrSynonymParser.parse(SolrSynonymParser.java:71) ~[lucene-analysis-common-9.12.0.jar:?] at org.elasticsearch.analysis.common.SynonymTokenFilterFactory.buildSynonyms(SynonymTokenFilterFactory.java:250) ~[?:?] at org.elasticsearch.analysis.common.SynonymTokenFilterFactory.getChainAwareTokenFilterFactory(SynonymTokenFilterFactory.java:197) ~[?:?] at org.elasticsearch.index.analysis.AnalyzerComponents.createComponents(AnalyzerComponents.java:83) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.analysis.ReloadableCustomAnalyzer.reload(ReloadableCustomAnalyzer.java:141) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.analysis.IndexAnalyzers$1.reload(IndexAnalyzers.java:162) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.mapper.MapperService.reloadSearchAnalyzers(MapperService.java:806) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.indices.IndicesService$3.beforeIndexShardRecovery(IndicesService.java:634) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.lambda$iterateBeforeIndexShardRecovery$12(CompositeIndexEventListener.java:303) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:269) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:300) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:327) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1784) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:133) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:421) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] ... 8 more 

among others. I can see also

[2024-10-09T10:59:38,410][WARN ][o.e.i.c.IndicesClusterStateService] [v8.16.0-3] [another_index][0] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [another_index][0]: Recovery failed on {v8.16.0-3}{fYAWdGI_RLypAvD-ll3rDQ}{nY0Sx-SrRm6EDAq8FdbWHA}{v8.16.0-3}{127.0.0.1}{127.0.0.1:40471}{cdfhilmrstw}{8.16.0}{7000099-8516000}{testattr=test, ml.allocated_processors_double=32.0, ml.machine_memory=126604152832, ml.config_version=12.0.0, ml.max_jvm_size=536870912, ml.allocated_processors=32, xpack.installed=true, transform.config_version=10.0.0} at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$36(IndexShard.java:3366) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$9(StoreRecovery.java:400) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$12(StoreRecovery.java:504) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:188) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:193) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:32) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:346) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:75) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:410) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:399) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:311) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:210) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:175) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:500) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:96) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2498) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway ... 24 more Caused by: org.elasticsearch.ResourceNotFoundException: synonyms set [set_missing] not found at org.elasticsearch.synonyms.SynonymsManagementAPIService.lambda$checkSynonymSetExists$15(SynonymsManagementAPIService.java:486) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.synonyms.SynonymsManagementAPIService$DelegatingIndexNotFoundActionListener.onResponse(SynonymsManagementAPIService.java:600) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1499) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:433) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] at org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390) ~[elasticsearch-8.16.0-SNAPSHOT.jar:?] ... 5 more 

@carlosdelest might you know what's the issue in the above? I cannot seem to reproduce it locally. Was the yml test supposed to be working on a yellow cluster (meaning some shards could not be allocated)?

[carlosdelest]

@kingherc , I think that is due to some of the tests done in rest-api-spec/test/synonyms/110_synonyms_invalid.yml . Those tests create an invalid synonym chain, which would fail when recovering index shards that have an invalid synonym.

That would cause the health of the cluster not to be green as the shards for the index are not recoverable.

This tests is meant for versions > 8.16.0, as specified by the cluster_features: ["gte_v8.16.0"]. If this is getting executed in bwc, it's expected to fail and leave the corresponding index in a red status, so the health check just introduced won't work.

Could we limit the cluster_health check to the index for the corresponding test, to avoid any other interaction with other indices that may not be in a green status?

[kingherc]

Thanks for the explanation @carlosdelest ! Pushed your recommendation to the PR, fingers crossed.

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x