Skip to content

[CI] GeoIpDownloaderIT testUseGeoIpProcessorWithDownloadedDBs failing with Message not fully read #96438

@pgomulka

Description

@pgomulka

I could not see any changes in the geoIP serialisaiton. It does not reproduce locally too

1> [2023-05-30T11:46:28,117][WARN ][o.e.t.TcpTransport ] [node_s0] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:35964, remoteAddress=127.0.0.1/127.0.0.1:19541, profile=default}], closing connection 1> java.lang.Exception: java.lang.AssertionError: java.lang.IllegalStateException: Message not fully read (response) for requestId [192], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.ActionListenerResponseHandler@94f41f2/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener/notifyOnce[[cluster:monitor/ingest/geoip/stats][{node_s1}{MYbBzhJ9SD6YeT3etp8llQ}{uPItn9j_Qv-nonD-byDXBw}{node_s1}{127.0.0.1}{127.0.0.1:19541}{cdfhilrstw}{8.9.0}]]/release[refCounted[org.elasticsearch.action.support.CancellableFanOut$$Lambda$3163/0x00000008019064e8@b3aa1b5]]], error [false]; resetting 1>	at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.exceptionCaught(Netty4MessageInboundHandler.java:75) ~[transport-netty4-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.92.Final.jar:4.1.92.Final] 1>	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.92.Final.jar:4.1.92.Final] 1>	at java.lang.Thread.run(Thread.java:1623) ~[?:?] 1> Caused by: java.lang.AssertionError: java.lang.IllegalStateException: Message not fully read (response) for requestId [192], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.ActionListenerResponseHandler@94f41f2/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener/notifyOnce[[cluster:monitor/ingest/geoip/stats][{node_s1}{MYbBzhJ9SD6YeT3etp8llQ}{uPItn9j_Qv-nonD-byDXBw}{node_s1}{127.0.0.1}{127.0.0.1:19541}{cdfhilrstw}{8.9.0}]]/release[refCounted[org.elasticsearch.action.support.CancellableFanOut$$Lambda$3163/0x00000008019064e8@b3aa1b5]]], error [false]; resetting 1>	at __randomizedtesting.SeedInfo.seed([BC7DEB88A745D747]:0) ~[?:?] 1>	at org.elasticsearch.transport.InboundHandler.verifyResponseReadFully(InboundHandler.java:190) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:418) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:368) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:95) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:821) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63) ~[transport-netty4-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	... 18 more 1> Caused by: java.lang.IllegalStateException: Message not fully read (response) for requestId [192], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.ActionListenerResponseHandler@94f41f2/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener/notifyOnce[[cluster:monitor/ingest/geoip/stats][{node_s1}{MYbBzhJ9SD6YeT3etp8llQ}{uPItn9j_Qv-nonD-byDXBw}{node_s1}{127.0.0.1}{127.0.0.1:19541}{cdfhilrstw}{8.9.0}]]/release[refCounted[org.elasticsearch.action.support.CancellableFanOut$$Lambda$3163/0x00000008019064e8@b3aa1b5]]], error [false]; resetting 1>	at org.elasticsearch.transport.InboundHandler.verifyResponseReadFully(InboundHandler.java:187) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:418) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:368) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:95) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:821) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86) ~[elasticsearch-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboun 1> dHandler.java:63) ~[transport-netty4-8.9.0-SNAPSHOT.jar:8.9.0-SNAPSHOT] 1>	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] 1>	... 18 more 

Build scan:
https://gradle-enterprise.elastic.co/s/2vgcyit5yqwwa/tests/:modules:ingest-geoip:internalClusterTest/org.elasticsearch.ingest.geoip.GeoIpDownloaderIT/testUseGeoIpProcessorWithDownloadedDBs

Reproduction line:

./gradlew ':modules:ingest-geoip:internalClusterTest' --tests "org.elasticsearch.ingest.geoip.GeoIpDownloaderIT.testUseGeoIpProcessorWithDownloadedDBs" -Dtests.seed=BC7DEB88A745D747 -Dtests.locale=da -Dtests.timezone=America/Araguaina -Druntime.java=20 

Applicable branches:
main

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.ingest.geoip.GeoIpDownloaderIT&tests.test=testUseGeoIpProcessorWithDownloadedDBs

Failure excerpt:

java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([BC7DEB88A745D747]:0) 

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions