vz: add SSH over AF_VSOCK

[norio-nomura]

Description:

Since systemd v256 (Ubuntu 24.10), SSH is bound to AF_VSOCK port 22.

https://github.com/systemd/systemd/releases/tag/v256

• If the system is run in a VM providing AF_VSOCK support, it automatically
  binds sshd to AF_VSOCK port 22.

https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878

• When sshd is installed on a system, a new systemd generator, systemd-ssh-generator
  binds a socket-activated SSH server to local AF_VSOCK and AF_UNIX sockets under certain conditions.

This PR changes to delay starting SSH port forwarding until the SSH server on the VM becomes ready. If AF_VSOCK port 22 can be connected, start a local SSH port as a proxy for AF_VSOCK port 22, instead of starting gvisor's port forwarder.

SSH over VSOCK is faster than SSH over gvisor's port forwarder.

This change is opt-out because it requires VZ and VM with systemd v256+,
setting LIMA_SSH_OVER_VSOCK=true does not mean it works.
To disable, set LIMA_SSH_OVER_VSOCK=false.

Benchmark logs:

On MacBook Pro 14 inch, 2023 with Apple M2 Pro

SETUP:

$ sw_vers ProductName:	macOS ProductVersion:	15.7 BuildVersion:	24G219 $ limactl start template://ubuntu-24.10 --rosetta --containerd=none --log-level error <snip> $ limactl shell ubuntu-24.10 sudo apt-get -U install -y iperf3 -qqq <snip>

GRPC Port Forwarder (Current):

$ limactl shell ubuntu-24.10 iperf3 -s WARN[0000] Both top-level 'rosetta' and 'vmOpts.vz.rosetta' are configured. Using vmOpts.vz.rosetta. Top-level 'rosetta' is deprecated.  ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from ::1, port 53668 [ 5] local ::1 port 5201 connected to ::1 port 53672 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 354 MBytes 2.97 Gbits/sec  [ 5] 1.00-2.00 sec 359 MBytes 3.01 Gbits/sec  [ 5] 2.00-3.00 sec 354 MBytes 2.97 Gbits/sec  [ 5] 3.00-4.00 sec 354 MBytes 2.97 Gbits/sec  [ 5] 4.00-5.00 sec 357 MBytes 3.00 Gbits/sec  [ 5] 5.00-6.00 sec 354 MBytes 2.97 Gbits/sec  [ 5] 6.00-7.00 sec 355 MBytes 2.98 Gbits/sec  [ 5] 7.00-8.00 sec 355 MBytes 2.98 Gbits/sec  [ 5] 8.00-9.00 sec 355 MBytes 2.98 Gbits/sec  [ 5] 9.00-9.98 sec 352 MBytes 3.01 Gbits/sec  - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-9.98 sec 3.47 GBytes 2.98 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated

SSH Port Forwarder on gvisor's virtual network (Old):

$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=false limactl restart ubuntu-24.10 --log-level error $ limactl shell ubuntu-24.10 iperf3 -s WARN[0000] Both top-level 'rosetta' and 'vmOpts.vz.rosetta' are configured. Using vmOpts.vz.rosetta. Top-level 'rosetta' is deprecated.  ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from ::1, port 43814 [ 5] local ::1 port 5201 connected to ::1 port 43824 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 277 MBytes 2.32 Gbits/sec  [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec  [ 5] 2.00-3.00 sec 278 MBytes 2.34 Gbits/sec  [ 5] 3.00-4.00 sec 279 MBytes 2.34 Gbits/sec  [ 5] 4.00-5.00 sec 279 MBytes 2.34 Gbits/sec  [ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec  [ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec  [ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec  [ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec  [ 5] 9.00-10.00 sec 279 MBytes 2.34 Gbits/sec  [ 5] 10.00-10.03 sec 9.00 MBytes 2.34 Gbits/sec  - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.03 sec 2.74 GBytes 2.34 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated

SSH Port Forwarder over AF_VSOCK (New):

$ LIMA_SSH_PORT_FORWARDER=true limactl restart ubuntu-24.10 --log-level error $ limactl shell ubuntu-24.10 iperf3 -s WARN[0000] Both top-level 'rosetta' and 'vmOpts.vz.rosetta' are configured. Using vmOpts.vz.rosetta. Top-level 'rosetta' is deprecated.  ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- Accepted connection from ::1, port 57942 [ 5] local ::1 port 5201 connected to ::1 port 57948 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 422 MBytes 3.54 Gbits/sec  [ 5] 1.00-2.00 sec 425 MBytes 3.57 Gbits/sec  [ 5] 2.00-3.00 sec 427 MBytes 3.58 Gbits/sec  [ 5] 3.00-4.00 sec 426 MBytes 3.58 Gbits/sec  [ 5] 4.00-5.00 sec 426 MBytes 3.58 Gbits/sec  [ 5] 5.00-6.00 sec 428 MBytes 3.59 Gbits/sec  [ 5] 6.00-7.00 sec 426 MBytes 3.58 Gbits/sec  [ 5] 7.00-8.00 sec 428 MBytes 3.59 Gbits/sec  [ 5] 8.00-9.00 sec 425 MBytes 3.57 Gbits/sec  [ 5] 9.00-10.00 sec 425 MBytes 3.57 Gbits/sec  [ 5] 10.00-10.00 sec 1.62 MBytes 3.83 Gbits/sec  - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 4.16 GBytes 3.57 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2) ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated

[AkihiroSuda]

Can be another PR, but wondering if we can further optimize the performance with (an equivalent of) systemd-ssh-proxy — SSH client plugin for connecting to AF_VSOCK and AF_UNIX sockets
https://www.freedesktop.org/software/systemd/man/256/systemd-ssh-proxy.html

[norio-nomura]

I don't understand the meaning of the comment.

[AkihiroSuda]

I thought it might be possible to eliminate the TCP->vsock forwarder and let ssh directly connect to the vsock.

[norio-nomura]

VM's VSOCK should not be open to the host unless the process that is starting the VM is mediated. As this PR is doing.

[norio-nomura]

Ubuntu 24.04 and earlier, SSH over AF_VSOCK can be enabled with:

provision: - mode: yq path: /etc/systemd/system/ssh.socket.d/vsock.conf format: ini expression: |  .Socket.ListenStream="vsock::22"

and upgraded SSH tracked at sshd socket activation does not support AF_VSOCK

[norio-nomura]

How to enable SSH over AF_VSOCK on Ubuntu 20.04, 22.04, and 24.04

Ubuntu 20.04, 22.04:

Since the patched version of SSH has already been released, it requires:

• Add ListenStream="vsock::22" to ssh.socket configuration
• Change ssh.service to use socket-based activation

$ limactl start template://ubuntu-20.04 --rosetta --containerd=none --set '.provision|=.//empty + [{  "mode": "yq",  "format": "ini",  "path": "/etc/systemd/system/ssh.socket.d/vsock.conf",  "expression": ".Socket.ListenStream=\"vsock::22\"" }, {  "mode": "system",  "script": ("#!/bin/bash set -eux -o pipefail systemctl is-enabled ssh.service || exit 0 # use socket based activation systemctl disable --now ssh.service systemctl enable --now ssh.socket "|. style="literal") }]' $ limactl restart ubuntu-20.04 2>&1 |grep -i vsock time="2025-09-07T17:01:40+09:00" level=info msg="[hostagent] started vsock forwarder: localhost:59291 -> vsock:22 on VM" time="2025-09-07T17:01:40+09:00" level=info msg="[hostagent] Detected SSH server is listening on the vsock port; changed localhost:59291 to proxy for the vsock port"

Ubuntu 24.04:

Since the patched version of SSH is not yet released (2025/09/07), it requires installing the proposed version.
The patched version of SSH has been released (2025/09/09), update openssh-server to latest release.

• Add ListenStream="vsock::22" to the ssh.socket configuration.
• Install the proposed version of SSH. Update openssh-server to patched version (1:9.6p1-3ubuntu13.14)
• Already, SSH has been changed to default socket-based activation.

$ limactl start template://ubuntu-24.04 --rosetta --containerd=none --set '.provision|=.//empty + [{  "mode": "yq",  "format": "ini",  "path": "/etc/systemd/system/ssh.socket.d/vsock.conf",  "expression": ".Socket.ListenStream=\"vsock::22\"" }, {  "mode": "system",  "script": ("#!/bin/bash ss -l --vsock|grep \*:22 -q && exit 0 apt-get --update install --assume-yes openssh-server "|. style="literal") }]' $ limactl restart ubuntu-24.04 2>&1 |grep -i vsock time="2025-09-07T17:01:56+09:00" level=info msg="[hostagent] started vsock forwarder: localhost:59299 -> vsock:22 on VM" time="2025-09-07T17:01:56+09:00" level=info msg="[hostagent] Detected SSH server is listening on the vsock port; changed localhost:59299 to proxy for the vsock port"

Edit: The patched version of openssh-server has been released on Ubuntu 24.04

[norio-nomura]

I'm considering another PR for the port forwarder implementation to VSOCK, which can be defined in lima.yaml, but I'm not sure if it's better to mix the rules with .portForwards[]. 🤔

[norio-nomura]

Ubuntu 24.04:

Since the patched version of SSH is not yet released (2025/09/07), it requires installing the proposed version.

The patched version of SSH has been released (2025/09/09), update openssh-server to latest release.

[norio-nomura]

To support custom usernet, I will move WaitOpeningSSHPort logic to the server side.

[norio-nomura]

To support custom usernet, I will move WaitOpeningSSHPort logic to the server side.

done.

[norio-nomura]

To support custom usernet, I will move WaitOpeningSSHPort logic to the server side.

This added an entry point /extension/wait_port?ip=<ip address>&port=<port> to endpoint sock of usernet.

[norio-nomura]

Updated to use SSHAddress as the host address.

[norio-nomura]

This line was not effective until this PR, because above defer:

filesToRemove := make(map[string]struct{}) defer func() { for f := range filesToRemove { _ = os.RemoveAll(f)	}	}()

has passed before filesToRemove[pidFile] = struct{}{} has effect.
Because I changed to wait waitSSHLocalPortAccessible, above defer changed to remove pidFile.
That’s why test fails with:
https://github.com/lima-vm/lima/actions/runs/17849303510/job/50754678943?pr=3979#step:9:165

time="2025-09-19T05:30:22Z" level=fatal msg="expected status "Running", got "Broken" (maybe use limactl stop -f?)"

I think filesToRemove is not required.
I'll remove them.

[AkihiroSuda]

Thanks

[norio-nomura]

Thanks! 🙏🏻