Unix socket vs TCP/IP host:port

Question

Could someone please describe to me the pros and cons of using a Unix socket file vs a tcp/ip localhost:port when setting up services on a server (Ubuntu, FWIW)?

In this particular instance it's for a Python WSGI server (uWSGI) but I'm just interested in general (eg, I know you can set up MySQL in both ways).

I realise that using tcp/ip means that the services can be exposed to other machines, but I'm just interested in whether there are any performance tradeoffs when accessing services locally.

Cheers.

Answer 1

Unix sockets are a little bit faster as you don't have the tcp-overhead. If you realize this performance loss is a question of server load. If you don't have very high server load you won't recognize it.

If you use Jails (FreeBSD) or some other virtualisation technology to separate the e.g. MySQL-Server from the Webserver, you often use the tcp/ip setup instead of sockets. The firewall rules need to restrict the access though.

You need to find out if your system is under heavy load so that a socket is a must or you can focus on a nice system design (separating services), then a tcp/ip solution would be better.

So make a long answer short:

Yes, there is a performance difference, sockets are faster. If you are not suffering high server load, just choose what fits better to your system's design.

Answer 2

It's basically a tradeoff between performance and flexibility. Unix domain sockets will give you a bit better performance, while a socket connected to localhost gives you a bit better portability. You can easily move the server app to another OS by just changing the IP address from localhost to a different hostname.

A Unix domain socket uses the local file system to create an IPC mechanism between the server and client processes. You will see a file in /var somewhere when the Unix domain socket is connected.

If you are looking for purely the ultimate performance solution you may want to explore a shared memory IPC. But, that's a little bit more complex.

Answer 3

Pros of Unix domain sockets.

1. Access can be managed through the Unix user permission system either by setting the permissions on the socket itself or by the server reading the usename of the connecting client.
2. Less chance of inadvertently exposing the socket to the outside world. For example if the server also runs a web proxy then that may inadvertently allow connections to sockets on localhost.

Cons of Unix domain sockets

1. Not portable to non-unix systems.
2. Can be awkward with chroots, jails or similar

Answer 4

On my machine, Ryzen 3900X, Ubuntu 22,

A basic C++ TCP server app that only sends 64K packets, and a basic c++ receiver that pulls 100GB of these packets and copies them to it's internal buffer only, single-threaded:

achieves ~30-33 GBit/sec for TCP connection (~4.0GB/sec) (not MBit)

and ~55-58GBit/sec for a socket connection, (~7.3 GB/sec)

and ~492Gbit/sec for in-process memcopy (~61GB /sec)

and, of course, would be faster still for a "shared memory" approach where you do not do any copies at all, merely pass a reference with some sort of synchronisation mechanism.

That's depending on whether I move the mouse and windows around during the test. So there clearly is a substantial difference in overhead, and could be in range of 2x depending on the details.

So for a high throughput, resource-constrained, embedded computer like Mosquito over Raspberry pi, there will be a benefit in using an unix socket and a benefit in not splitting the overall application into modules too small.

Of course, if your application only ever serves a database search replies or a web site, and your response time is in multiple millisecond range, then the processing time will dominate the network latency. But, for tiny computers that are stressed to the peak, and e.g. use multiple micro-services or multiple processing steps, this might make a difference in the overall application latency.

When developing for ultimate performance, it is worth knowing that the Linux console output is relatively costly -- so if in benchmark/production mode you can do away with printing to console, you may get a substantial performance boost.