4

I'm attempting a gateway hop through servers A -> B -> C, and would like to have this connection as an entry in my SSH config.

SSHing from A into B works. And then from B to C works. As does ssh B -t ssh C.

However, when I attempt use the following SSH config file, it fails.

Host B Hostname B Host C Hostname C ProxyJump B Host * User username ForwardAgent yes PKCS11Provider /usr/lib/ssh-keychain.dylib

When running this verbosely, I find I'm running into a problem with

debug1: getpeername failed: Bad file descriptor

This answer seems to suggest that problem arises from the lookup for C not being found (namely, inside the /etc/hosts file). When I look at the contents of /etc/hosts on B, all the final host locations I would like to connect to (including C) are listed. So I believe I want my connection to use B's /etc/hosts listing when making the final connection. Is there a way I can specify this in A's SSH config?

Note, I do not have root permissions on any of the machines (A, B, nor C).

Debug log:

username@A ~ % ssh C -v OpenSSH_7.9p1, LibreSSL 2.7.3 debug1: Reading configuration data /Users/username/.ssh/config debug1: /Users/username/.ssh/config line 14: Applying options for C debug1: /Users/username/.ssh/config line 19: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug1: Setting implicit ProxyCommand from ProxyJump: ssh -v -W '[%h]:%p' B debug1: Executing proxy command: exec ssh -v -W '[C]:22' B OpenSSH_7.9p1, LibreSSL 2.7.3 debug1: Reading configuration data /Users/username/.ssh/config debug1: /Users/username/.ssh/config line 19: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug1: Connecting to B [111.111.111.111] port 22. debug1: Connection established. debug1: provider /usr/lib/ssh-keychain.dylib: manufacturerID <Apple, Inc.> cryptokiVersion 2.20 libraryDescription <Keychain emulation PKCS#11 API> libraryVersion 0.0 debug1: provider /usr/lib/ssh-keychain.dylib slot 0: label <Key For PIV Authentication (Use> manufacturerID <Apple, Inc.> model <Keychain> serial <000000> flags 0x404 debug1: have 1 keys debug1: provider /usr/lib/ssh-keychain.dylib slot 1: label <Key For Digital Signature (User> manufacturerID <Apple, Inc.> model <Keychain> serial <000000> flags 0x404 debug1: have 2 keys debug1: identity file /Users/username/.ssh/id_rsa type -1 debug1: identity file /Users/username/.ssh/id_rsa-cert type -1 debug1: identity file /Users/username/.ssh/id_dsa type -1 debug1: identity file /Users/username/.ssh/id_dsa-cert type -1 debug1: identity file /Users/username/.ssh/id_ecdsa type -1 debug1: identity file /Users/username/.ssh/id_ecdsa-cert type -1 debug1: identity file /Users/username/.ssh/id_ed25519 type -1 debug1: identity file /Users/username/.ssh/id_ed25519-cert type -1 debug1: identity file /Users/username/.ssh/id_xmss type -1 debug1: identity file /Users/username/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.9 debug1: provider /usr/lib/ssh-keychain.dylib: manufacturerID <Apple, Inc.> cryptokiVersion 2.20 libraryDescription <Keychain emulation PKCS#11 API> libraryVersion 0.0 debug1: provider /usr/lib/ssh-keychain.dylib slot 0: label <Key For PIV Authentication (Use> manufacturerID <Apple, Inc.> model <Keychain> serial <000000> flags 0x404 debug1: have 1 keys debug1: provider /usr/lib/ssh-keychain.dylib slot 1: label <Key For Digital Signature (User> manufacturerID <Apple, Inc.> model <Keychain> serial <000000> flags 0x404 debug1: have 2 keys debug1: identity file /Users/username/.ssh/id_rsa type -1 debug1: identity file /Users/username/.ssh/id_rsa-cert type -1 debug1: identity file /Users/username/.ssh/id_dsa type -1 debug1: identity file /Users/username/.ssh/id_dsa-cert type -1 debug1: identity file /Users/username/.ssh/id_ecdsa type -1 debug1: identity file /Users/username/.ssh/id_ecdsa-cert type -1 debug1: identity file /Users/username/.ssh/id_ed25519 type -1 debug1: identity file /Users/username/.ssh/id_ed25519-cert type -1 debug1: identity file /Users/username/.ssh/id_xmss type -1 debug1: identity file /Users/username/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.9 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4 debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002 debug1: Authenticating to B:22 as 'username' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:XXXXXXXXXXXXXXXXXXXXXXX debug1: Host 'B' is known and matches the ECDSA host key. debug1: Found key in /Users/username/.ssh/known_hosts:5 debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 4294967296 blocks debug1: Will attempt key: /usr/lib/ssh-keychain.dylib RSA SHA256:YYYYYYYYYYYYYYYYYYYYYYYYY token agent debug1: Will attempt key: /usr/lib/ssh-keychain.dylib RSA SHA256:ZZZZZZZZZZZZZZZZZZZZZZZZZ token agent debug1: Will attempt key: /Users/username/.ssh/id_rsa debug1: Will attempt key: /Users/username/.ssh/id_dsa debug1: Will attempt key: /Users/username/.ssh/id_ecdsa debug1: Will attempt key: /Users/username/.ssh/id_ed25519 debug1: Will attempt key: /Users/username/.ssh/id_xmss debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive debug1: Next authentication method: publickey debug1: Offering public key: /usr/lib/ssh-keychain.dylib RSA SHA256:YYYYYYYYYYYYYYYYYYYYYYYYY token agent debug1: Server accepts key: /usr/lib/ssh-keychain.dylib RSA SHA256:YYYYYYYYYYYYYYYYYYYYYYYYY token agent debug1: pkcs11_provider_unref: 0x111111111111 refcount 3 debug1: pkcs11_provider_unref: 0x111111111111 refcount 2 debug1: Authentication succeeded (publickey). Authenticated to B ([111.111.111.111]:22). debug1: channel_connect_stdio_fwd C:22 debug1: channel 0: new [stdio-forward] debug1: getpeername failed: Bad file descriptor debug1: Requesting [email protected] debug1: Entering interactive session. debug1: pledge: exec debug1: client_input_global_request: rtype [email protected] want_reply 0 debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc execution disabled. debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc execution disabled. channel 0: open failed: administratively prohibited: open failed stdio forwarding failed ssh_exchange_identification: Connection closed by remote host

/etc/ssh/ssh_config

Host * SendEnv LANG LC_* ForwardX11 yes ForwardX11Trusted yes XAuthLocation /opt/X11/bin/xauth
Improve this question
17
  • Try using ProxyCommand instead of JumpProxy. See this article for more info. Commented Feb 10, 2020 at 7:46
  • @harrymc, I've attempted ProxyCommand ssh B -W %h:%p as an alternative to JumpProxy B, but this results in the same issue. Are you recommending any specific variation of the ProxyCommand parameters? Commented Feb 10, 2020 at 16:50
  • 1
    More information may help. Try adding a debug log here using ssh -v. Commented Feb 10, 2020 at 16:56
  • @harrymc, added the debug log. Commented Feb 10, 2020 at 23:28
  • 1
    Do you mean that you have tried the fixes in the post SSH tunneling error and they didn't help? Could you add to the post the files /etc/ssh/ssh_config and ~/.ssh/config (if it exists)? Commented Feb 12, 2020 at 7:17

7 Answers 7

Reset to default
1

When you connect to any of the hosts, the lookup is done on your localhost, not remote, so I would suggest to forget /etc/hosts and be sure to have HostName with the proper IP/DNS for host C.

Also, are you using for jump:

 ProxyCommand ssh -W %h:%p B

I noticed you were using it the other way around (in your comments)

Improve this answer
5
  • Do you have any specific recommendations for how to obtain the "proper IP/DNS"? I have attempted the each of fully resolved names and the IP for C in B's /etc/hosts (by using them as the Hostname in A's config). As for the ProxyCommand argument order, I don't believe it should have an impact, but I tried the swapped ordering (as listed in your post) and received the same error. Commented Feb 12, 2020 at 19:24
  • my command is different, I'm using ProxyCommand which seems like the old way to do it; you show you want to use ProxyJump although you have JumpProxy (which is wrong). So either ProxyJump or ProxyCommand should work; also, let's says that: ` - HostA = 10.10.10.1 - HostB = 10.10.10.2 - HostC = 10.10.10.3 ` you want to reach HostC via Host B, just use ` Host B HostName 10.10.10.2 Host C HostName 10.10.10.3 ProxyJump HostB ` this should work; because the "proper IP" of C is being read on host A, and passed as an argument to host B, and then jump to host C. Commented Feb 17, 2020 at 17:08
  • Sorry, JumpProxy was just a typo in my example config file for the question. I have been using ProxyJump in the real case. I also attempted ProxyCommand as you noted (with both orderings of the parameters, and using both IP addresses or domain name host names). The issue given in the question occurs in all cases. Commented Feb 17, 2020 at 17:20
  • I have other hosts with intermediate gateway servers configured identically to the above, except with different targets and gateway servers (e.g., A -> D -> E). In these cases, the configuration successfully connects. In these cases, both the ProxyCommand and ProxyJump options work. It is due to the specific configuration of the host resolution with B and C that there is an issue, and this is what I'm looking for a solution to. Commented Feb 17, 2020 at 17:29
  • Just to note, we can see in the debug log that the ProxyCommand/ProxyJump is successfully connecting to the appropriate intermediate host. It is only during the resolution of C from B that there is an issue. Commented Feb 17, 2020 at 17:40
1
+50

In the lack of any other explanation, I believe that the problem is that you are missing a reverse dns entry, perhaps via a line in /etc/hosts.

I would suggest updating the /etc/hosts on all the three computers, so they all know about each other.

Improve this answer
1
  • Without permissions to /etc/hosts this may be difficult. However, with this method of specifying a user level hosts file it might be doable. Unfortunately, I won’t be back at my work station until after the bounty grace period. But as it’s most likely the correct solution, and the rest of your effort walking through the options (in the comments), I will be giving this answer the bounty. Commented Feb 18, 2020 at 1:02
0

I have the following config running here:
~/.ssh/config (of my_host)

Host C HostName 192.168.3.1 # ip of Host C ProxyCommand ssh -W %h:%p B ServerAliveInterval 60 Host B HostName 192.168.2.1 # ip of Host B ProxyCommand ssh -W %h:%p A ServerAliveInterval 60 Host A HostName 192.168.1.1 # ip of Host A ServerAliveInterval 60

This provides for a connection like this:
'my_host' -> 'Host A' -> 'Host B' -> 'Host C'

If you use DNS-Names instead of the ip-addresses, then you need to make sure that each host can resolve the DNS-Name of the next host.
-> your host can resolve the DNS-Name of Host A
-> Host A can resolve the DNS-Name of Host B
-> Host B can resolve the DNS-Name of Host C

If you only need one hop and Host A of your example is your work station, then the following /ssh/config of Host A should do: 

Host C HostName 192.168.3.1 # ip of Host C ProxyCommand ssh -W %h:%p B ServerAliveInterval 60 Host B HostName 192.168.2.1 # ip of Host B ServerAliveInterval 60
Improve this answer
2
  • A is the local machine. A can resolve B (during the first SSH command from A). B can resolve C (in the second SSH command from B). The problem arises when executing the single command with the Jump or ProxyCommand. Replacing the target the Hostname with the IP address results in the same issue. Commented Feb 17, 2020 at 2:28
  • Can you pls. move PKCS11Provider /usr/... from Host * to Host B and try again? Commented Feb 17, 2020 at 17:46
0

After some research, I've to change my answer.

I think your problem doesn't have anything to do with
debug1: getpeername failed: Bad file descriptor
as this line also appears in my working configuration. 

debug1: channel_connect_stdio_fwd target:22 debug1: channel 0: new [stdio-forward] debug1: getpeername failed: Bad file descriptor debug1: Requesting [email protected] debug1: Entering interactive session. debug1: pledge: network debug1: client_input_global_request: rtype [email protected] want_reply 0

I'm not sure yet of the 'pledge: network' vs. 'pledge: exec'

Looking at the messages to the end of log

debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc execution disabled. debug1: Remote: Agent forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: User rc execution disabled.

What happens if you comment out ForwardAgent yes in ~/.ssh/config and ForwardX11 yes, ForwardX11Trusted yes as well as XAuthLocation /opt/X11/bin/xauth in _/etc/ssh/ssh_config_?

Improve this answer
4
  • I do not have root access, and so cannot edit /etc/ssh/ssh_config. However, by overriding these settings in my own config file using ForwardAgent no, ForwardX11 no, ForwardX11Trusted no, and XAuthLocation /usr/local/bin/xauth, then I get the same result, but with pledge: network instead of pledge: exec. Commented Feb 19, 2020 at 21:53
  • 1
    "SSHing from B into A works." Is that really from B to A? "And then from B to C works." After you SSHed from B to A? That sounds like B -> A -> C Also it would be interesting to know what configuration your user on B uses to connect to C. If I remember correctly then on a forwarded connection all config is done on the first system. Therefore if B can connect to C and A can not connect to C via B, then either B doesn't allow a forwarded connection or there is a difference between the config on A and B. Commented Feb 20, 2020 at 7:00
  • Ah, sorry! That was a typo (I'm slightly amazed at how many typos I had in this question). This should say "SSHing from A into B works". I have corrected it now. Thank you for pointing this out. Commented Feb 20, 2020 at 16:09
  • Then, if it works from A to B and from B to C but not fromA to C via B, we should compare the ssh config and parameters used by A and B. Maybe that shed a light on the problem. Commented Feb 20, 2020 at 16:59
0

I had this exact problem, as well.

In my case, name resolution worked fine exactly as expected.

The cause turned out to be that TCP port forwarding was disabled on the intermediate hosts. The solution thus was to change AllowTcpForwarding to yes in the intermediate host's /etc/ssh/sshd_config

In my case, I had several Match clauses in sshd_config file, so I had to change the AllowTcpForwarding in multiple places.

Improve this answer
0

Here is the problem:

debug1: Remote: Port forwarding disabled.

According to ssh man page -J / ProxyJump works in the following way:

Connect to the target host by first making a ssh connection to the jump host described by destination and then establishing a TCP forwarding to the ultimate destination from there.

This basically means that the jump server must allow TCP forwarding for this option to work which is not the case here.

One possible workaround would be to use socat or netcat on the jump host to forward the connection to target host. Something along the lines:

Host C Hostname C ProxyCommand ssh B socat STDIO TCP:%h:%p
Improve this answer
0

In lack of root access, you can use only ssh config. Why not set C's IP in the ssh config on A?

Host B Hostname B Host C Hostname <ip of C> JumpProxy B Host * User username ForwardAgent yes PKCS11Provider /usr/lib/ssh-keychain.dylib
Improve this answer
1
  • I have attempted using the IP of C in place of the name of C, but this results in the same issue. Commented Feb 17, 2020 at 2:22

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.