[CI] DocsClientYamlTestSuiteIT test {yaml=reference/ccr/apis/get-ccr-stats} #35496
Description
Failure:
DocsClientYamlTestSuiteIT.test {yaml=reference/ccr/apis/get-ccr-stats/line_88} <<< FAILURES! > Throwable #1: java.lang.AssertionError: Failure at [reference/ccr/apis/get-ccr-stats:74]: $body didn't match expected value: > $body: > auto_follow_stats: > number_of_failed_follow_indices: same [0] > number_of_failed_remote_cluster_state_requests: expected [0] but was [1] > number_of_successful_follow_indices: same [0] > recent_auto_follow_errors: > received [1] more entries than expected > follow_stats: > indices: > 0: > index: same [follower_index] > shards: > 0: > bytes_read: same [0] > failed_read_requests: same [0] > failed_write_requests: same [0] > follower_global_checkpoint: same [0] > follower_index: same [follower_index] > follower_mapping_version: same [0] > follower_max_seq_no: same [0] > follower_settings_version: same [0] > last_requested_seq_no: same [0] > leader_global_checkpoint: same [0] > leader_index: same [leader_index] > leader_max_seq_no: same [0] > operations_read: same [0] > operations_written: same [0] > outstanding_read_requests: same [0] > outstanding_write_requests: same [0] > read_exceptions: same [empty list] > remote_cluster: same [remote_cluster] > shard_id: same [0] > successful_read_requests: same [0] > successful_write_requests: same [0] > time_since_last_read_millis: same [-1] > total_read_remote_exec_time_millis: same [0] > total_read_time_millis: same [0] > total_write_time_millis: same [0] > write_buffer_operation_count: same [0] > write_buffer_size_in_bytes: same [0] > at __randomizedtesting.SeedInfo.seed([9F632EDCF44DF Does not reproduce. Log is full with connection error. The AutoFollowCoordinator could fetch remote cluster state, because the license couldn't be determined and that was caused by a node disconnect exception:
[2018-11-13T06:31:16,896][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node-0] failure occurred while fetching cluster state for auto follow pattern [my_auto_follow_pattern] org.elasticsearch.ElasticsearchStatusException: can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined at org.elasticsearch.xpack.ccr.CcrLicenseChecker.clusterStateUnknownRemoteLicense(CcrLicenseChecker.java:425) ~[x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.xpack.ccr.CcrLicenseChecker.lambda$checkRemoteClusterLicenseAndFetchClusterState$5(CcrLicenseChecker.java:171) ~[x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.xpack.ccr.CcrLicenseChecker$1.onFailure(CcrLicenseChecker.java:217) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onFailure(RemoteClusterLicenseChecker.java:180) [x-pack-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1127) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.transport.TransportService.lambda$onConnectionClosed$8(TransportService.java:982) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.ElasticsearchException: could not determine the license type for cluster [remote_cluster] ... 9 more Caused by: org.elasticsearch.transport.NodeDisconnectedException: [node-0][127.0.0.1:59722][cluster:monitor/xpack/info] disconnected ... SKIPPED 56 LINES ... The test failed because a the auto follow stats are not expected to report failures (number_of_failed_remote_cluster_state_requests should be 0 and recent_auto_follow_errors should be empty).
Returned auto follow stats:
{ 1> "auto_follow_stats" : { 1> "number_of_failed_follow_indices" : 0, 1> "number_of_failed_remote_cluster_state_requests" : 1, 1> "number_of_successful_follow_indices" : 0, 1> "recent_auto_follow_errors" : [ 1> { 1> "leader_index" : "my_auto_follow_pattern", 1> "auto_follow_exception" : { 1> "type" : "exception", 1> "reason" : "ElasticsearchStatusException[can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined]; nested: ElasticsearchException [could not determine the license type for cluster [remote_cluster]]; nested: NodeDisconnectedException[[node-0][127.0.0.1:59722][cluster:monitor/xpack/info] disconnected];", 1> "caused_by" : { 1> "type" : "status_exception", 1> "reason" : "can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined", 1> "caused_by" : { 1> "type" : "exception", 1> "reason" : "could not determine the license type for cluster [remote_cluster]", 1> "caused_by" : { 1> "type" : "node_disconnected_exception", 1> "reason" : "[node-0][127.0.0.1:59722][cluster:monitor/xpack/info] disconnected" 1> } 1> } 1> } 1> } 1> } 1> ] 1> }