1

Summary

I want to copy files between 2 remote servers using Rsync and SSH. The way I found to do this is creating a SSH tunnel, but I tried with several examples and any of those worked.

Details

  • I have 3 machines: 1 local computer and 2 remote servers
  • I can SSH to both remote servers using my local RSA key.
  • The 2 remote servers cannot talk to each other. Remote #1 does not have access to Remote #2 and viceversa.
  • I can Rsync to both remote servers individually

What I have tried

ssh -vvv -A \ -l user_remote1 \ -R localhost:50000:remote2:22 user_remote1@remote1 \ 'rsync -e \ "ssh -A -o StrictHostKeyChecking=no \ -o UserKnownHostsFile=/dev/null \ -o GlobalKnownHostsFile=/dev/null \ -p 50000" \ -var /home/user_remote1/test.txt user_remote2@localhost:/home/user_remote2'

Output of that command

OpenSSH_8.1p1, LibreSSL 2.7.3 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 47: Applying options for * debug2: resolve_canonicalize: hostname remote1 is address debug2: ssh_connect_direct debug1: Connecting to remote1 [remote1] port 22. debug1: Connection established. debug1: identity file /Users/user_local/.ssh/id_rsa type 0 debug1: identity file /Users/user_local/.ssh/id_rsa-cert type -1 debug1: identity file /Users/user_local/.ssh/id_dsa type -1 debug1: identity file /Users/user_local/.ssh/id_dsa-cert type -1 debug1: identity file /Users/user_local/.ssh/id_ecdsa type -1 debug1: identity file /Users/user_local/.ssh/id_ecdsa-cert type -1 debug1: identity file /Users/user_local/.ssh/id_ed25519 type -1 debug1: identity file /Users/user_local/.ssh/id_ed25519-cert type -1 debug1: identity file /Users/user_local/.ssh/id_xmss type -1 debug1: identity file /Users/user_local/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0 debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to remote1:22 as 'user_remote1' debug3: hostkeys_foreach: reading file "/Users/user_local/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /Users/user_local/.ssh/known_hosts:4 debug3: load_hostkeys: loaded 1 keys from remote1 debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512 debug2: compression ctos: none,[email protected] debug2: compression stoc: none,[email protected] debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:6gPivXbLGMtZsqEIwH4713sH1S/p/pQfHJ8+DoTsqc8 debug3: hostkeys_foreach: reading file "/Users/user_local/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /Users/user_local/.ssh/known_hosts:4 debug3: load_hostkeys: loaded 1 keys from remote1 debug1: Host 'remote1' is known and matches the ECDSA host key. debug1: Found key in /Users/user_local/.ssh/known_hosts:4 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /Users/user_local/.ssh/id_rsa RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o debug1: Will attempt key: /Users/user_local/.ssh/id_dsa debug1: Will attempt key: /Users/user_local/.ssh/id_ecdsa debug1: Will attempt key: /Users/user_local/.ssh/id_ed25519 debug1: Will attempt key: /Users/user_local/.ssh/id_xmss debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /Users/user_local/.ssh/id_rsa RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: /Users/user_local/.ssh/id_rsa RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o debug3: sign_and_send_pubkey: RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o debug3: sign_and_send_pubkey: signing using rsa-sha2-512 debug3: send packet: type 50 debug3: receive packet: type 52 debug1: Authentication succeeded (publickey). Authenticated to remote1 ([remote1]:22). debug1: Remote connections from localhost:50000 forwarded to local address remote2:22 debug3: send packet: type 80 debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug3: send packet: type 90 debug1: Requesting [email protected] debug3: send packet: type 80 debug1: Entering interactive session. debug1: pledge: network debug3: receive packet: type 80 debug1: client_input_global_request: rtype [email protected] want_reply 0 debug3: receive packet: type 4 debug1: Remote: /home/user_remote1/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug3: receive packet: type 4 debug1: Remote: /home/user_remote1/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding debug3: receive packet: type 81 debug1: remote forward success for: listen localhost:50000, connect remote2:22 debug1: All remote forwarding requests processed debug3: receive packet: type 91 debug2: channel_input_open_confirmation: channel 0: callback start debug1: Requesting authentication agent forwarding. debug2: channel 0: request [email protected] confirm 0 debug3: send packet: type 98 debug2: fd 3 setting TCP_NODELAY debug3: ssh_packet_set_tos: set IP_TOS 0x20 debug2: client_session2_setup: id 0 debug1: Sending environment. debug3: Ignored env TERM_PROGRAM debug3: Ignored env TERM debug3: Ignored env SHELL debug3: Ignored env TMPDIR debug3: Ignored env TERM_PROGRAM_VERSION debug3: Ignored env OLDPWD debug3: Ignored env TERM_SESSION_ID debug3: Ignored env USER debug3: Ignored env SSH_AUTH_SOCK debug3: Ignored env PATH debug3: Ignored env PWD debug3: Ignored env XPC_FLAGS debug3: Ignored env XPC_SERVICE_NAME debug3: Ignored env HOME debug3: Ignored env SHLVL debug3: Ignored env LOGNAME debug1: Sending env LC_CTYPE = UTF-8 debug2: channel 0: request env confirm 0 debug3: send packet: type 98 debug3: Ignored env _ debug3: Ignored env __CF_USER_TEXT_ENCODING debug1: Sending command: rsync -e \\ "ssh -A -o StrictHostKeyChecking=no \\ -o UserKnownHostsFile=/dev/null \\ -o GlobalKnownHostsFile=/dev/null \\ -p 50000" \\ -var /home/user_remote1/test.txt user_remote2@localhost:/home/user_remote2 debug2: channel 0: request exec confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: exec request accepted on channel 0 debug3: receive packet: type 90 debug1: client_input_channel_open: ctype forwarded-tcpip rchan 4 win 2097152 max 32768 debug1: client_request_forwarded_tcpip: listen localhost port 50000, originator ::1 port 56418 debug2: fd 7 setting O_NONBLOCK debug2: fd 7 setting TCP_NODELAY debug1: connect_next: host remote2 ([remote2]:22) in progress, fd=7 debug3: fd 7 is O_NONBLOCK debug3: fd 7 is O_NONBLOCK debug1: channel 1: new [::1] debug1: confirm forwarded-tcpip debug3: channel 1: waiting for connection debug1: channel 1: connected to remote2 port 22 debug3: send packet: type 91 debug3: receive packet: type 90 debug1: client_input_channel_open: ctype [email protected] rchan 5 win 65536 max 16384 debug2: fd 8 setting O_NONBLOCK debug3: fd 8 is O_NONBLOCK debug1: channel 2: new [authentication agent connection] debug1: confirm [email protected] debug3: send packet: type 91 debug2: channel 0: rcvd ext data 84 Warning: Permanently added '[localhost]:50000' (ECDSA) to the list of known hosts. debug2: channel 0: written 84 to efd 6 debug3: receive packet: type 96 debug2: channel 2: rcvd eof debug2: channel 2: output open -> drain debug2: channel 2: obuf empty debug2: channel 2: chan_shutdown_write (i0 o1 sock 8 wfd 8 efd -1 [closed]) debug2: channel 2: output drain -> closed debug1: channel 2: FORCE input drain debug2: channel 2: ibuf empty debug2: channel 2: send eof debug3: send packet: type 96 debug2: channel 2: input drain -> closed debug2: channel 2: send close debug3: send packet: type 97 debug3: channel 2: will not send data after close debug3: receive packet: type 97 debug2: channel 2: rcvd close debug3: channel 2: will not send data after close debug2: channel 2: is dead debug2: channel 2: garbage collecting debug1: channel 2: free: authentication agent connection, nchannels 3 debug3: channel 2: status: The following connections are open: #0 client-session (t4 r2 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1) #1 ::1 (t4 r4 i0/0 o0/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1) #2 authentication agent connection (t4 r5 i3/0 o3/0 e[closed]/0 fd 8/8/-1 sock 8 cc -1) debug2: channel 0: rcvd ext data 38 Permission denied, please try again. debug2: channel 0: written 38 to efd 6 debug2: channel 0: rcvd ext data 38 Permission denied, please try again. debug2: channel 0: written 38 to efd 6 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0 debug2: channel 0: rcvd eow debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 6 [write]) debug2: channel 0: input open -> closed debug2: channel 0: rcvd ext data 229 debug3: receive packet: type 96 debug2: channel 1: rcvd eof debug2: channel 1: output open -> drain debug2: channel 1: obuf empty debug2: channel 1: chan_shutdown_write (i0 o1 sock 7 wfd 7 efd -1 [closed]) debug2: channel 1: output drain -> closed debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: obuf_empty delayed efd 6/(229) user_remote2@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3] debug2: channel 0: written 229 to efd 6 debug3: channel 0: will not send data after close debug2: channel 0: obuf empty debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 5 efd 6 [write]) debug2: channel 0: output drain -> closed debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 2 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r2 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1) #1 ::1 (t4 r4 i0/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1) debug2: channel 1: read<=0 rfd 7 len 0 debug2: channel 1: read failed debug2: channel 1: chan_shutdown_read (i0 o3 sock 7 wfd 7 efd -1 [closed]) debug2: channel 1: input open -> drain debug2: channel 1: ibuf empty debug2: channel 1: send eof debug3: send packet: type 96 debug2: channel 1: input drain -> closed debug2: channel 1: send close debug3: send packet: type 97 debug3: channel 1: will not send data after close debug3: receive packet: type 97 debug2: channel 1: rcvd close debug3: channel 1: will not send data after close debug2: channel 1: is dead debug2: channel 1: garbage collecting debug1: channel 1: free: ::1, nchannels 1 debug3: channel 1: status: The following connections are open: #1 ::1 (t4 r4 i3/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1) debug3: send packet: type 1 debug3: fd 1 is not O_NONBLOCK Transferred: sent 6168, received 6820 bytes, in 2.5 seconds Bytes per second: sent 2488.0, received 2750.9 debug1: Exit status 255

I have searched about several related topics from that output like searching about "rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]", "ssh debug1: Next authentication method: password" or "read_passphrase: can't open /dev/tty: No such device or address", but could not find anything useful.

Is there something wrong with the command or maybe I need a custom configuration in the servers? Any feedback will be useful. Thanks.

Improve this question
4
  • See Rsync equivalent to scp -3. If that answer doesn't work for you, Google something like "scp -3" for rsync Commented Mar 18, 2021 at 21:40
  • Have a look on this post, where the SSH command is a bit different than yours. Commented Mar 18, 2021 at 21:41
  • @bitinerant I got the same response "Host key verification failed. rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]". Unfortunately scp is not an option. Commented Mar 18, 2021 at 22:33
  • @harrymc that was one of the examples I did, but didn't work. Commented Mar 18, 2021 at 22:35

1 Answer 1

Reset to default
0

You're using ssh -A. I understand you want ssh run by rsync on remote1 to reach your local ssh-agent. You're hoping to get authorized on remote2 with your local key this way. The plan is good.

The verbose output you published tells me ssh-agent is working locally and ssh run by rsync on remote1 can access it. Still you get Permission denied (publickey,…. The (tunneled) SSH connection from remote1 to remote2 fails to authenticate and closes; rsync cannot continue (rsync: connection unexpectedly closed).

This strongly suggests the local agent does not hold the right key. man 1 ssh-agent reads:

The agent initially does not have any private keys. Keys are added using ssh-add(1) or by ssh(1) when AddKeysToAgent is set in ssh_config(5).

One way to add a key to an already started ssh-agent is with ssh-add before you invoke ssh. Since you want to use the same key your local ssh is going to use, AddKeysToAgent should work as well. You don't need to modify ssh_config, you can pass the option directly in the (local) ssh invocation:

ssh -A -o AddKeysToAgent=yes …

(where … indicates the rest of the command you already have).

Note this requires ssh-agent to run and SSH_AUTH_SOCK to be in the environment, so ssh can communicate with the agent. It seems you have already taken care of this.

Users with similar problem who know little about ssh-agent and how to start it properly (or they are not sure if ssh-agent has been started properly by their OS, by themselves or whatever) may prefer the following command:

ssh-agent ssh -A -o AddKeysToAgent=yes …

It starts a separate (new) ssh-agent. The agent will start ssh in the right environment, so the two can communicate. There are scenarios where this form is particularly useful (example). Here the form is not a must, however it's independent of any agent previously started (or not started at all) and should work out of the box, unless there is some problem with the key(s). So if you're not familiar with ssh-agent then the last command is the best choice. ssh-agent started this way will terminate after ssh exits.

In the OP's case there are few minor improvements to make:

  • ssh -l user_remote1 user_remote1@remote1 is redundant, it specifies user_remote1 twice. There are few methods to specify the user and the server. Using more than one method in a single invocation is hardly ever useful (see an edge case when it's kinda useful).

  • I think nothing on remote2 needs to access the local ssh-agent. -A in the ssh command rsync (on remote1) is going to use is therefore superfluous. The command will work either way, but mind what man 1 ssh states:

    Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.

    You need -A when you ssh from local to remote1; an attacker sitting on remote1 is the risk you take. The superfluous -A when connecting to remote2 is only an opportunity for an attacker sitting on remote2.

Improve this answer

You must log in to answer this question.