[CI] SSLException: Received close_notify during handshake #34514
Description
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+multijob-windows-compatibility/64/consoleFull has failed with:
12:42:32 1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received close_notify during handshake 12:42:32 1> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] 12:42:32 1> Caused by: javax.net.ssl.SSLException: Received close_notify during handshake 12:42:32 1> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?] 12:42:32 1> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?] 12:42:32 1> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?] 12:42:32 1> at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1776) ~[?:?] 12:42:32 1> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?] 12:42:32 1> at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?] 12:42:32 1> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?] 12:42:32 1> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_181] 12:42:32 1> at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final] 12:42:32 1> ... 15 more reproduction line of the build above with build id `20181016071459-6169C3F9 (does not reproduce locally):
gradlew :x-pack:qa:tribe-tests-with-security:test \ -Dtests.seed=E96200A29197A00B \ -Dtests.class=org.elasticsearch.xpack.security.SecurityTribeTests \ -Dtests.security.manager=true \ -Dtests.locale=de-LU \ -Dtests.timezone=Australia/Tasmania reproduction line of build id `20181008232140-2FF07565 (does not reproduce locally):
./gradlew :x-pack:plugin:security:test \ -Dtests.seed=103C6D419C2C4438 \ -Dtests.class=org.elasticsearch.xpack.security.transport.netty4.SimpleSecurityNetty4ServerTransportTests \ -Dtests.method="testSendRandomRequests" \ -Dtests.security.manager=true \ -Dtests.locale=sq \ -Dtests.timezone=MST7MDT \ -Dcompiler.java=11 \ -Druntime.java=8 reproduction line of build id 20181007194343-B582AB07 (does not reproduce locally):
/gradlew :x-pack:qa:tribe-tests-with-security:test \ -Dtests.seed=E256452AFB8FDE46 \ -Dtests.class=org.elasticsearch.xpack.security.SecurityTribeTests \ -Dtests.method="testRetrieveRolesOnTribeNode" \ -Dtests.security.manager=true \ -Dtests.locale=lv-LV \ -Dtests.timezone=Africa/Nouakchott So far we have the following build ids that failed with this error:
| build id | JDK version | OS / Distro | branch |
|---|---|---|---|
| 20181016071459-6169C3F9 | 10+46 | Windows 2012 | 6.4 |
| 20181010113501-73ECBC82 | 10+46 | Oracle Linux 7 | 6.4 |
| 20181008232140-2FF07565 | 8 | Ubuntu 16.04 | master |
| 20181007194343-B582AB07 | 10+46 | CentOS 7 | 6.4 |
This exception is raised when a sender sends close_notify to the recipient to indicate it will not send
any more messages on this connection (see also RFC5246). The question is why this happens during the SSL handshake.
I wonder whether this could be caused by a similar issue in Netty as addressed in #30337 although I did not find anything related in recent Netty tickets. From the mix of failures on different operating systems I think we can rule out the OS.