Linux automatically timeouts incoming connections just after accepting

Question

We have a dedicated server that works fine and accepts incoming connections fine. it receives and sends data fine as well.

Works great for some hours, but after that all accepted connections automatically time out just after the socket is accepted.

The more it accepts sockets, the sooner it goes into a state which all new connections will time out.

The problem is not from client, and is not for really the time we set in timeout, because it just happens after accepting in read() function.

When we restart the server, all things start working great again.

I believe linux is causing this problem. I don't know linux well, I know there are configs in linux which are limiting my process.

Can someone suggest material to read for linux configs for tcp sockets?

thanks

Answer 1

Could you please provide more details? What kind of service accepts connections? What distro is used (please, provide output of uname -a)? Do you have any info in service logs?

You can check system logs with the following commands systemctl status <service> to see service status and few lines of logs.

And journalctl -x --lines 100 -u <service> to see last 100 lines of service logs.

Also command dmesg or files like /var/log/syslog and /var/log/kern.log in case of debian-like system may contain additional info. Or /var/log/messages in case of rhel-like.

The issue also may be related to number of open files or service configuration (like connections limit in database configuration or max number of threads in tomcat). Usually modern systems don't need modification of default settings. But some old systems may fail to open new connections if the limit is reached. https://linux.die.net/man/5/limits.conf

Kernel settings related to networking: https://docs.kernel.org/networking/ip-sysctl.html

Parameters might be configured for all interfaces or per interface using appropriate path. Settings might be changed on-the-fly with two ways (for example, disabling ipv6):

1. echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6 - disables ipv6 for all interfaces until reboot
2. sysctl -w net.ipv6.conf.ens160.disable_ipv6=0 >> /etc/sysctl.d/sysctl.conf - disables ipv6 for interface ens160 and adds appropriate line at the end of /etc/sysctl.d/sysctl.conf. In this case settings will be applied on next reboot.

Useful commands which also may help: lsof to list open files (connection is also counted as file) and ss or netstat.

For example lsof -i:8090 will list all connection to port 8090. And lsof -p 2345 will list all files and connection open by process with PID 2345.

Please note that before making any changes in kernel parameters it's better to know the real cause.

Answer 2

after months of struggling with this I found the root problem.

I closed fd 0 which is stdin and new incoming sockets would pick fd 0 and in my code I was expecting any valid fd is over 0, otherwise means already closed or must be evicted from valid connections (and I do close that fd to make sure :D before evicting).

and that was causing fd 0 to always be closed after picking it.

that was my bad.