10

I am trying to connect to github over ssh on my remote server (Running Ubuntu 22.04).

On my local computer (Running Win 10), I have ~/.ssh/config file with the following:

Host remote HostName SERVER_IP port 22 User ubuntu ForwardAgent yes

After connecting to the remote server, I can confirm that the ssh agent is working by typing:

echo "$SSH_AUTH_SOCK"

result: /tmp/ssh-XXXXPWEKZo/agent.1073

Also with ssh-add -l I see that the key is added

4096 SHA256:hvGuLtIuwYi2LAnQ0KdC/9IgdBUmlHZer0NyXUXd5aY C:\Users\user/.ssh/id_rsa (RSA)

In /etc/ssh/sshd_config I have allowed Agent forwarding with AllowAgentForwarding yes

But when I try to connect to github ssh -T [email protected] -vvv (the key is added to github settings) I get the following:

OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/ubuntu/.ssh/known_hosts' debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/ubuntu/.ssh/known_hosts2' debug2: resolving "github.com" port 22 debug3: resolve_host: lookup github.com:22 debug3: ssh_connect_direct: entering debug1: Connecting to github.com [140.82.121.3] port 22. debug3: set_sock_tos: set socket 3 IP_TOS 0x10 debug1: Connection established. debug1: identity file /home/ubuntu/.ssh/id_rsa type -1 debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1 debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/ubuntu/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1 debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_ed25519_sk type -1 debug1: identity file /home/ubuntu/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_xmss type -1 debug1: identity file /home/ubuntu/.ssh/id_xmss-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_dsa type -1 debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3 debug1: Remote protocol version 2.0, remote software version babeld-2f5f2727 debug1: compat_banner: no match: babeld-2f5f2727 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to github.com:22 as 'git' debug3: record_hostkey: found key type ED25519 in file /home/ubuntu/.ssh/known_hosts:1 debug3: load_hostkeys_file: loaded 1 keys from github.com debug1: load_hostkeys: fopen /home/ubuntu/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,[email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256 debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256 debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr debug2: ciphers stoc: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr debug2: MACs ctos: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256 debug2: MACs stoc: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: SSH2_MSG_KEX_ECDH_REPLY received debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU debug3: record_hostkey: found key type ED25519 in file /home/ubuntu/.ssh/known_hosts:1 debug3: load_hostkeys_file: loaded 1 keys from github.com debug1: load_hostkeys: fopen /home/ubuntu/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: Host 'github.com' is known and matches the ED25519 host key. debug1: Found key in /home/ubuntu/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: ssh_set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: ssh_set_newkeys: mode 0 debug1: rekey in after 134217728 blocks channel 1: chan_shutdown_read: shutdown() failed for fd 7 [i0 o0]: Not a socket debug2: get_agent_identities: ssh_agent_bind_hostkey: communication with agent failed debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed debug1: Will attempt key: /home/ubuntu/.ssh/id_rsa debug1: Will attempt key: /home/ubuntu/.ssh/id_ecdsa debug1: Will attempt key: /home/ubuntu/.ssh/id_ecdsa_sk debug1: Will attempt key: /home/ubuntu/.ssh/id_ed25519 debug1: Will attempt key: /home/ubuntu/.ssh/id_ed25519_sk debug1: Will attempt key: /home/ubuntu/.ssh/id_xmss debug1: Will attempt key: /home/ubuntu/.ssh/id_dsa debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/ubuntu/.ssh/id_rsa debug3: no such identity: /home/ubuntu/.ssh/id_rsa: No such file or directory debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa_sk debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa_sk: No such file or directory debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519 debug3: no such identity: /home/ubuntu/.ssh/id_ed25519: No such file or directory debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519_sk debug3: no such identity: /home/ubuntu/.ssh/id_ed25519_sk: No such file or directory debug1: Trying private key: /home/ubuntu/.ssh/id_xmss debug3: no such identity: /home/ubuntu/.ssh/id_xmss: No such file or directory debug1: Trying private key: /home/ubuntu/.ssh/id_dsa debug3: no such identity: /home/ubuntu/.ssh/id_dsa: No such file or directory debug2: we did not send a packet, disable method debug1: No more authentication methods to try. [email protected]: Permission denied (publickey).

I don't have such problems on other of my servers that are running ubuntu 20.04 and 18.04.

Improve this question
4
  • If you see the key with ssh-add -l on the remote server then your issue is not with agent forwarding. However your ssh logs show that it fails to communicate with the agent, have you run all these commands in order in the same terminal and shell on the remote machine? Please post the full ssh -vvv to have more debugging information. Commented Oct 23, 2022 at 18:56
  • Hello, I have edited the question with the full -vvv output. On the remote server if I use the same key, but without forwarding (I manually upload it to the server) everything works Ok. Commented Oct 24, 2022 at 5:42
  • I just noticed while testing with the same key but without forwarding, that even if I add the key to the agent on the remote server, when connecting to github it says debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed and it asks for the key password, so the problem should be with the agent on the remote server. It's a clean install of Ubuntu 22.04 provided by ovh.com Commented Oct 24, 2022 at 6:00
  • 1
    I have the same issue, and it seems to be connected to the ssh-client on Windows. When I try a forward with an agent running on a linux, it works fine. Commented Nov 24, 2022 at 12:07

4 Answers 4

Reset to default
11

This answer helped me solve the same problem: there's some incompatibility between the ssh client shipped with Windows and the server on Ubuntu 22.04.

tldr: Installing the most recent release of the windows ssh client fixed the issue for me.

Improve this answer
3
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review Commented Dec 4, 2022 at 12:48
  • Note: do not just download the MSI file and click it, wishing everything will be done. Instead, RTFM here: github.com/PowerShell/Win32-OpenSSH/wiki/… Commented May 12, 2023 at 18:02
  • Note: the release link is stale, most recent can be found at the top of this page: github.com/PowerShell/Win32-OpenSSH/releases Commented Oct 24, 2023 at 14:52
3

This is already answered but If you are here to look for a solution for an error like chan_shutdown_read: channel 1: shutdown() failed for fd 7 [i0 o0]: Not a socket

This only applies if you have installed latest version of OpenSSH from GitHub and your forwarding agent suddenly stopped.

In my case after Windows update Forwarding Agent suddenly stopped working. I could connect the server but since my SSH key is not forwarded I could not able to push or pull from GitHub Private Repo using SSH and It was saying access denied for public key.

To confirm if the solution is same: Open services.msc and find OpenSSH Authentication Agent. Double click to open it's properties and check Path to executable section. If it is not "C:\Program Files\OpenSSH\ssh-agent.exe" and pointing to ssh agent in system32 than this solution will work.

open regedit and head to HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/ssh-agent Change ImagePath value to "C:\Program Files\OpenSSH\ssh-agent.exe" and restart the computer.

After restart make sure that OpenSSH Authentication Agent service is running then add your SSH key by typing ssh-add in PowerShell or CMD. Then your Forwarding should work.

Improve this answer
2
  • and restart the computer. FWIW, Restart-Service ssh-agent also seems to work. It even keeps the keys you had in there (ssh-add -L to check). Commented Jun 18, 2024 at 13:50
  • Most of the time just updating the regedit value and re-opening the app that is using SSH is enough. Commented Jul 1, 2024 at 6:17
2

The answer from @kaytwo is right; updating the Windows ssh client is the cure.

Here's what strace -tt -f -v -s1024 ssh $hostname revealed:

2636504 12:05:25.582068 write(3, "\0\0\0\320", 4) = 4 2636504 12:05:25.582159 write(3, "\33\0\0\0\[email protected]\0\0\0003\0\0\0\vssh-ed25519\0\0\0.....binary-data-omitted.....", 208) = 208 2636504 12:05:25.582210 read(3, "", 4) = 0 2636504 12:05:25.875257 getpid() = 2636504 2636504 12:05:25.875343 write(3, "\0\0\0\1", 4) = 4 2636504 12:05:25.875532 write(3, "\v", 1) = 1 2636504 12:05:25.875691 read(3, "", 4) = 0 2636504 12:05:25.875745 getpid() = 2636504 2636504 12:05:25.875799 write(2, "debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed\r\n", 87) = 87

Compare with strace -tt -f -v -s1024 ssh-add -l:

2648504 13:00:38.631807 connect(3, {sa_family=AF_UNIX, sun_path="/tmp/ssh-XXXXuWGsZN/agent.2628678"}, 110) = 0 2648504 13:00:38.632285 write(3, "\0\0\0\1", 4) = 4 2648504 13:00:38.632386 write(3, "\v", 1) = 1 2648504 13:00:38.632464 read(3, "\0\0\1\255", 4) = 4 2648504 13:00:38.920854 read(3, "\f\0\0\0\1\0\0\1\227\0\0\0\7ssh-rsa\0\0\0\3\1\0\1\0\0\1...binary-data-omitted.....", 429) = 429

Basically, sending a "[email protected]" request causes the Windows ssh-agent to stop talking; the normal "list all keys" command doesn't return anything when it's in that state. Fortunately, the state is cleared when the connection is closed, so you can try again without having to restart anything.

It looks like this configuration option (in man ssh_config) is related:

 PubkeyAuthentication Specifies whether to try public key authentication. The argument to this keyword must be yes (the default), no, unbound or host-bound. The final two options enable public key authentication while respectively disabling or enabling the OpenSSH host-bound au‐ thentication protocol extension required for restricted ssh-agent(1) forwarding.

Note: While that paragraph says ssh -o PubkeyAuthentication=unbound ..... should disable this protocol extension, strace showed that it was still being sent, and the Windows ssh agent won't work.

This page on SSH Agent Restriction goes into total detail about the subject, from the reasons that inspired the feature to command-line options needed to use it and the byte sequences sent in the SSH Agent protocol extension.

Good luck!

Improve this answer
2
  • 1
    This seems to be my issue, too. It is possible to identify this cause by enabling LogLevel DEBUG3 on the server you are connecting to (/etc/ssh/sshd_config). It shows as active: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding and followed by a session_auth_agent_req: agent forwarding disabled in the logs. The last is trigged from this code line if (auth_restricted(RESTRICT_AGENT, s->pw)) { }, which is explained here: openssh.com/agent-restrict.html (your link). Commented Aug 22, 2024 at 7:06
  • Also, check the ssh server versions with ssh -V. The issue with missing agent forwarding only appear on servers with OpenSSH_9.6p1 (Debian 12, Ubuntu 24.0.4), not on servers with OpenSSH_8.9p1 (e.g. Ubuntu 22.04). Commented Aug 22, 2024 at 7:22
1

For me the problem was gpg-agent-ssh.socket was running (in user space).

I fixed Agent Forwarding on Ubuntu 24.04 Noble by executing:

systemctl --user disable --now gpg-agent-ssh.socket exit

and logging in using SSH again.

When GPG took over the SSH agent socket, it looked like this:

$ echo $SSH_AUTH_SOCK /run/user/1000/gnupg/S.gpg-agent.ssh

When normal SSH agent forwarding is working, it looks like this:

echo $SSH_AUTH_SOCK /tmp/ssh-RAJUTI1zbK/agent.127999
Improve this answer
3
  • Just for the sake of curiosity... do you have lsof installed? If so, what does lsof /run/user/1000/gnupg/S.gpg-agent.ssh say, compared to lsof /tmp/ssh-RAJUTI1zbK/agent.127999? My question is just to try to figure out how exactly the GnuPG agent is interfacing (or interfering!) with the OpenSSH agent. Commented Mar 14 at 16:42
  • lsof does not show anything, it would only show for a short moment when OpenSSH is using Agent Forwarding to connect. The value of SSH_AUTH_SOCK determines which socket is used. Commented Mar 15 at 20:50
  • Thank you for the clarification! Aye, you're correct, of course. Commented Mar 17 at 11:01

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.