Skip to content

Conversation

@masseyke
Copy link
Member

We don't currently make a defensive copy of the inputs to the GeoIpDownloaderStatsAction$NodeResponse constructor. Some of those inputs are subject to change when a geoip database is reloaded. That can cause serialization to break (because the serialization code assumes that the contents of the Sets are not changing during serialization. This PR makes an immutable copy of the Sets in NodeResponse to prevent this.
Before this fix, we would occasionally see failures like this:

Caused by: java.lang.IllegalStateException: Message not fully read (response) for requestId [192], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.ActionListenerResponseHandler@94f41f2/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener/notifyOnce[[cluster:monitor/ingest/geoip/stats][{node_s1}{MYbBzhJ9SD6YeT3etp8llQ}{uPItn9j_Qv-nonD-byDXBw}{node_s1}{127.0.0.1}{127.0.0.1:19541}{cdfhilrstw}{8.9.0}]]/release[refCounted[org.elasticsearch.action.support.CancellableFanOut$$Lambda$3163/0x00000008019064e8@b3aa1b5]]], error [false]; resetting 

I was able to force this to reproduce reliably by artificially putting a 2-second sleep into the serialization code (StreamOutput.writeCollection) between when the size of the collection is read and when the values are written. With this fix I have been unable to reproduce it.

Closes #96438

@masseyke masseyke added >bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v8.9.0 labels Jun 12, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Jun 12, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@elasticsearchmachine
Copy link
Collaborator

Hi @masseyke, I've created a changelog YAML for you.

Copy link
Member

@jbaiera jbaiera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice catch!

@masseyke masseyke merged commit 65fc5e5 into elastic:main Jun 12, 2023
@masseyke masseyke deleted the fix/GeoIpDownloaderStatsAction-NodeResponse-serialization branch June 12, 2023 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team v8.9.0

3 participants