1

I am downloading very slowly on some servers we host when using HTTPS on debian 8.1 kernel 3.16.0-4-amd64 Apache 2.4.10-10

The architecture is : CLIENT X.X.X.X => NAT ON FIREWALL: Y.Y.Y.Y => HTTP(S) WEB SERVER LAN IP:10.254.248.101

Test with HTTPS : 

wget --no-check-certificate https://Y.Y.Y.Y/test/debian-8.5.0-amd64-netinst.iso debian-8.5.0-amd64-netinst.iso.5 0%[ ] 632.00K 15.5KB/s eta 4h 30m

When running tcpdump trace on server (trace below), the client X.X.X.X ask 10.254.248.101 to resend packet many times. The server wait almost 4 seconds to resend the packet asked by the client. This is why I am downloading so slowly on the server. What I don't understand is why the server take a so long time to resend the packet. Is there something new in linux Kernel 3 for TCP ? Something new in Apache 2.4 ? Someting that FIREWALL is not able to manage ?

Note that, We don't have this problem in these cases :

  • When using HTTP (without TLS) (test below).
  • When clients come from ISP that have longer latency (200 to 250 ms) to our datacenter.
  • With same architecture, same SSL certificate but older linux kernel 2.6 and older apache 2.2.16-6.
  • When we put WEB SERVER before firewall set with pulic IP : CLIENT X.X.X.X => Y.Y.Y.Y.

Test with HTTP :

wget http://Y.Y.Y.Y//test/debian-8.5.0-amd64-netinst.iso debian-8.5.0-amd64-netinst.iso.6 35%[=============================> ] 86.53M 1.34MB/s eta 2m 2s

Trace of the serveur :

366 7.930488 X.X.X.X 10.254.248.101 66 TCP 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 369 7.931293 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#1] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546404710 370 7.933094 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#2] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546406098 371 7.933984 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#3] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546407486 372 7.934031 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#4] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546408874 373 7.935729 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#5] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546403322 SRE=3546410262 374 7.936154 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#6] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546403322 SRE=3546411650 375 7.937320 X.X.X.X 10.254.248.101 78 TCP [TCP Dup ACK 366#7] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546403322 SRE=3546413038 376 7.938613 X.X.X.X 10.254.248.101 86 TCP [TCP Dup ACK 366#8] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546415814 SLE=3546403322 SRE=3546413038 377 7.940425 X.X.X.X 10.254.248.101 86 TCP [TCP Dup ACK 366#9] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546417202 SLE=3546403322 SRE=3546413038 378 7.941661 X.X.X.X 10.254.248.101 86 TCP [TCP Dup ACK 366#10] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546418590 SLE=3546403322 SRE=3546413038 379 7.942561 X.X.X.X 10.254.248.101 86 TCP [TCP Dup ACK 366#11] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546419978 SLE=3546403322 SRE=3546413038 380 7.943155 X.X.X.X 10.254.248.101 86 TCP [TCP Dup ACK 366#12] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546421366 SLE=3546403322 SRE=3546413038 381 7.945145 X.X.X.X 10.254.248.101 86 TCP [TCP Dup ACK 366#13] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 382 7.945992 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#14] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546424142 SRE=3546425530 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 383 7.946974 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#15] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546424142 SRE=3546426918 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 384 7.948476 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#16] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546428306 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 385 7.949704 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#17] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546429694 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 386 7.950628 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#18] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546431082 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 387 7.951890 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#19] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546432470 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 388 7.953365 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#20] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546433858 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 389 7.954028 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#21] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546435246 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 390 7.955327 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#22] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546436634 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 391 7.956139 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#23] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546438022 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 392 7.957504 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#24] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546439410 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 393 7.958271 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#25] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546440798 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 394 7.963017 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#26] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038 395 7.963080 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#27] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546444962 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 396 7.963091 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#28] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546446350 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 397 7.963127 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#29] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546447738 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 398 7.963912 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#30] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546449126 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 399 7.965634 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#31] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546450514 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 400 7.966653 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#32] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546451902 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 401 7.967402 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#33] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546453290 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 402 7.968499 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#34] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546454678 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 403 7.969984 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#35] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546456066 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 404 7.971168 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#36] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546457454 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 405 7.972479 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#37] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 406 7.973633 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#38] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546460230 SRE=3546461618 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186 407 7.974609 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#39] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546460230 SRE=3546463006 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186 408 7.975828 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#40] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186 409 7.976604 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#41] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546467170 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 410 7.977806 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#42] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546468558 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 411 7.979200 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#43] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546469946 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 412 7.982792 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#44] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546471334 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 413 7.982817 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#45] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546472722 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 414 7.983331 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#46] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546474110 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 415 7.983631 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#47] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546475498 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 416 7.984371 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#48] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546476886 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 417 7.986238 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#49] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546478274 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 418 7.987165 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#50] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546479662 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 419 7.988157 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#51] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546481050 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 421 8.113745 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#52] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668420 TSecr=44657604 SLE=3546465782 SRE=3546482438 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 424 8.150664 X.X.X.X 10.254.248.101 60 TCP 65259 > 443 [ACK] Seq=1251 Ack=2197 Win=65700 Len=0 428 8.373938 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#54] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668446 TSecr=44657604 SLE=3546465782 SRE=3546485214 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 434 8.504768 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#55] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668459 TSecr=44657604 SLE=3546465782 SRE=3546486602 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 436 8.630589 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#56] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668472 TSecr=44657604 SLE=3546465782 SRE=3546487990 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 439 8.759603 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#57] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668485 TSecr=44657604 SLE=3546465782 SRE=3546489378 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 441 8.886688 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#58] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668498 TSecr=44657604 SLE=3546465782 SRE=3546490766 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 443 9.015359 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#59] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668510 TSecr=44657604 SLE=3546465782 SRE=3546492154 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 445 9.141981 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#60] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668523 TSecr=44657604 SLE=3546465782 SRE=3546493542 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 447 9.271283 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#61] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668536 TSecr=44657604 SLE=3546465782 SRE=3546494930 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 449 9.398242 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#62] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668549 TSecr=44657604 SLE=3546465782 SRE=3546496318 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 451 9.525714 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#63] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668562 TSecr=44657604 SLE=3546465782 SRE=3546497706 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 453 9.653846 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#64] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668574 TSecr=44657604 SLE=3546465782 SRE=3546499094 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 455 9.782376 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#65] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668587 TSecr=44657604 SLE=3546465782 SRE=3546500482 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 457 9.909855 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#66] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668600 TSecr=44657604 SLE=3546465782 SRE=3546501870 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 459 10.038509 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#67] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668613 TSecr=44657604 SLE=3546465782 SRE=3546503258 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 461 10.166752 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#68] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668626 TSecr=44657604 SLE=3546465782 SRE=3546504646 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 467 10.303645 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#69] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668639 TSecr=44657604 SLE=3546465782 SRE=3546506034 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 474 10.430023 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#70] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668652 TSecr=44657604 SLE=3546465782 SRE=3546507422 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 476 10.557951 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#71] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668665 TSecr=44657604 SLE=3546465782 SRE=3546508810 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 478 10.689920 X.X.X.X 10.254.248.101 94 TCP [TCP Dup ACK 366#72] 33426 > 443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668678 TSecr=44657604 SLE=3546507422 SRE=3546508810 SLE=3546465782 SRE=3546508810 SLE=3546460230 SRE=3546464394 482 11.045636 X.X.X.X 62.61.231.82 1454 TCP [TCP Retransmission] 443 > 33426 [ACK] Seq=223341 Ack=831 Win=31104 Len=1388 TSval=44658396 TSecr=94668678
Improve this question
1
  • Would you be able to email me a copy of the client and server captures, I'd like to take a look over them. Email is [email protected] Commented Aug 27, 2016 at 10:03

1 Answer 1

Reset to default
4

Thanks for sending those captures over.

The Problem

Your throughput issues appear to be caused by a buggy implementation of TCP Sequence Number randomization. I have seen this in the past on Cisco ASAs.

To give a bit of background, it was observed in the past that some TCP implementations did not use enough randomness when choosing an Initial Sequence Number (ISN) which made it easier for attackers to manipulate TCP connections by making educated guesses at what the Sequence number would be.

To attempt to fix this issue, some firewall providers implemented a feature called TCP sequence number randomization, which rewrites the Sequence number (SEQ) to a more random value, when it sees TCP packets flowing through the firewall. Unfortunately some implementations of this feature are a bit buggy and do not account for TCPs Selective Acknowledgement (SACK) feature.

You can see Sequence Number randomization in action in your trace. Look at the SYN/ACK packet from the server (packet #51 server capture), where you can see that the ISN chosen is 2847541373. However look at the same SYN/ACK packet when it is received on the client side (packet #8 client capture), the ISN has been changed to 2098751282!

This behavior is fine up until the point that packet loss is experienced on the network.

On the client side, look at the first Duplicate Acknowledgement (Dup ACK) at packet 259. You can see that a SACK block has been set covering bytes 2098977399-2098978787. This packet effectively tells the server, I'm waiting on packet with SEQ 2098974623, however I have received 2098977399-2098978787 so you don't need to send those again.

Now, if you look at the same Dup ACK as it is received on the server side (#369), you can see the ACK number has been correctly converted by the firewall (2098974623 > 2847764714), however the SACK block hasn't and still shows 2098977399-2098978787!

When a Dup ACK is received with an invalid SACK block, the Dup ACK is ignored.

As a result, you lose out on the ability to use Fast Retransmission (retransmit after 3 duplicate ACKs received) and rely solely on Retranmission Timeouts. This is really, really bad for performance and will reduce your throughput substantially.

So what can you do?

You can investigate whether TCP Sequence Number randomisation is still required for your purposes and if not, consider testing with it disabled. Perhaps this issue has been resolved in a newer firmware?

You could also turn off the TCP SACK option on your server(s) to prevent clients from using SACK in the first place /proc/sys/net/ipv4/tcp_sack however please note that SACK is meant to be used to improve TCP performance and the actual issue is with the firewalls (buggy) implementation of Sequence number randomization. Turning off SACK will mean that Dup ACK's from clients will no longer be ignored and the connection will be able to recover from loss a lot quicker. Throughput should go up.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.