6

I am trying to login to final_host from localhost via an intermediate hop, i.e:

localhost -> hop -> final_host

This works: 

localhost:~$ ssh -t user@hop "ssh user@final_host"

This also works:

localhost:~$ ssh user@hop hop:~$ ssh user@final_host finalhost:~$

But this doesn't:

localhost:~$ ssh -J user@hop_ip "ssh user@final_host_ip" -vvv OpenSSH_7.6p1, LibreSSL 2.6.2 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l user -vvv -W '[%h]:%p' hop_ip debug1: Executing proxy command: exec ssh -l user -vvv -W '[final_host_ip]:22' hop_ip debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6 debug1: permanently_drop_suid: 501 OpenSSH_7.6p1, LibreSSL 2.6.2 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to hop_ip port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000 debug3: fd 5 is O_NONBLOCK debug1: Authenticating to hop_ip:22 as 'user' debug3: hostkeys_foreach: reading file "/Users/user/.ssh/known_hosts" debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from hop_ip debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],ssh-ed25519 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: [email protected],ssh-ed25519,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-ed25519 debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected] debug2: compression stoc: none,[email protected] debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: [email protected] debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ssh-ed25519 SHA256:c18UMgj7nokTZJHMGnbsOgxDHlIZc2r184efDHtoTLE debug3: hostkeys_foreach: reading file "/Users/user/.ssh/known_hosts" debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from hop_ip debug1: Host 'hop_ip' is known and matches the ED25519 host key. debug1: Found key in /Users/user/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey after 134217728 blocks debug2: key: server_key_folder/.ssh/id_rsa (0x7f953cc00d70), agent debug2: key: /Users/user/.ssh/id_rsa (0x0) debug2: key: /Users/user/.ssh/id_dsa (0x0) debug2: key: /Users/user/.ssh/id_ecdsa (0x0) debug2: key: /Users/user/.ssh/id_ed25519 (0x0) debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: RSA SHA256:IhOfM2s2i/vFgY/Mj962CoNez631HDIMDRjxFvDhOEI server_key_folder/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 debug2: input_userauth_pk_ok: fp SHA256:IhOfM2s2i/vFgY/Mj963CoNez631HDIMDRvxFvDhOEI debug3: sign_and_send_pubkey: RSA SHA256:IhOfM2s2i/vFgY/Mj963CoNez631HDIMDRvxFvDhOEI debug3: send packet: type 50 debug3: receive packet: type 52 debug1: Authentication succeeded (publickey). Authenticated to hop_ip ([hop_ip]:22). debug3: ssh_init_stdio_forwarding: final_host_ip:22 debug1: channel_connect_stdio_fwd final_host_ip:22 debug1: channel 0: new [stdio-forward] debug2: fd 7 setting O_NONBLOCK debug2: fd 8 setting O_NONBLOCK debug1: getpeername failed: Bad file descriptor debug3: send packet: type 90 debug2: fd 5 setting TCP_NODELAY debug3: ssh_packet_set_tos: set IP_TOS 0x10 debug1: Requesting [email protected] debug3: send packet: type 80 debug1: Entering interactive session. debug1: pledge: network debug3: receive packet: type 80 debug1: client_input_global_request: rtype [email protected] want_reply 0 debug3: receive packet: type 91 debug2: channel_input_open_confirmation: channel 0: callback start debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 2097152 rmax 32768 debug1: ssh_exchange_identification: \033]1337;PushKeyLabels=fish_%self\033]1337;SetKeyLabel=F2=GitSSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 debug1: ssh_exchange_identification: debug1: ssh_exchange_identification: \024n��3�\030d\206\206�\235A6 debug1: ssh_exchange_identification: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug1: ssh_exchange_identification: sh.com,[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype [email protected] reply 1 debug3: send packet: type 100 debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug1: channel 0: FORCE input drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug1: stdio forwarding: done ssh_exchange_identification: Connection closed by remote host

It gets stuck for a while before closing the connection. Using a config file with ProxyJump, ProxyCommand -W or ProxyCommand + nc also doesn't work and gets stuck at the same point. Additionally this stopped working at some point, I was able to use ProxyCommand in the past and it suddenly stopped working one day without apparent config change from my side. What could be going on here and what can I do to debug further?

EDIT: I should point out that other users of the same system are able to login without issue via config file so the problem is most likely local.

Improve this question
5
  • 1
    What does type -a ssh show for you on the local system? Are you using some sort of wrapper for ssh? Can you test with a different hop server? Commented Feb 10, 2018 at 14:23
  • ssh is /usr/bin/ssh. I am not using any wrapper. Unfortunately can't test with a different hop (the hop is managed by someone else). Commented Feb 10, 2018 at 17:04
  • I would suggest checking the permissions on your ~/.ssh directory - should not be world-readable, but instead be 700 (r/w/x only by owner) Commented Feb 12, 2018 at 19:51
  • @eggo, thanks. I checked my permissions and the issue persists. Commented Feb 13, 2018 at 21:14
  • The debug log contains ssh_exchange_identification: \033]1337;PushKeyLabels which looks like some kind of escape sequence to emit some text. Are you sure your .bashrc or some other config doesn't mess with your Jump host environment? Commented Sep 25, 2020 at 12:17

2 Answers 2

Reset to default
2

Authentication to final_host succeeds, but you get this error: getpeername failed: Bad file descriptor.

The ssh daemon on the final_host and hop should connect file descriptors for you to use (stdin, stdout, stderr) on your localhost. This fails.

The function mentioned should point you in the right direction. You are using an IP address that might not have a reverse dns entry, nor a line in /etc/hosts.

Improve this answer
4
  • thanks, can you clarify how to get to a resolution based on your answer? Commented Feb 13, 2018 at 21:15
  • Based on above answer, it can be fixed by adding mapping for hop and final_host in hosts file Commented Feb 19, 2018 at 7:54
  • thanks for your comment, I added hop and final_host to my local /etc/hosts file and that didn't make a difference. Commented Feb 19, 2018 at 12:26
  • @bbaassssiiee, Where can you see that authenthification to final_host succeeds? Commented Feb 19, 2018 at 21:21
0

I had the same problem using the fish shell on macOS. A related ServerFault answer also mentioned fish and the iTerm2 shell integration, however removing and updating it made no difference for me.

In the end I discovered that I had another integration I created for auto-sourcing Python virtualenvs that was causing the problem - when I started a new shell without sourcing that the hanging did not occur.

For reference, the contents of my auto-source-venv.fish was

if test -e $VIRTUAL_ENV source $VIRTUAL_ENV/bin/activate.fish end

Which seems simple enough, and I'm not sure why SSH would be setting the $VIRTUAL_ENV environment variable. However, as the fish docs say, when testing the variable should be wrapped in double quotes - and indeed, after replacing the first line with if test -e "$VIRTUAL_ENV", my SSH ProxyJump is no longer hanging and connects as expected.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.