-
Pl
chevron_right
How to rsync files between two remotes?
pubsub.slavino.sk / planetdebian · Saturday, 6 November, 2021 - 10:41 · 3 minutes
scp -3
can copy files between two remote hosts through localhost.
This comes in handy when the two servers cannot communicate
directly or if they are unable to authenticate one to the
other.
1
Unfortunately,
rsync
does not support such a feature.
Here is a trick to emulate the behavior of
scp -3
with SSH tunnels.
When syncing with a remote host,
rsync
invokes
ssh
to spawn a
remote
rsync --server
process. It interacts with it through its
standard input and output. The idea is to recreate the same setup
using SSH tunnels and
socat
, a versatile tool to establish
bidirectional data transfers.
The first step is to connect to the source server and ask
rsync
the
command-line to spawn the remote
rsync --server
process. The
-e
flag overrides the command to use to get a remote shell: instead of
ssh
, we use
echo
.
$ ssh web04 $ rsync -e 'sh -c ">&2 echo $@" echo' -aLv /data/. web05:/data/. web05 rsync --server -vlogDtpre.iLsfxCIvu . /data/.rsync: connection unexpectedly closed (0 bytes received so far) [sender]rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]
The second step is to connect to the destination server with
local
port forwarding. When connecting to the local port 5000, the TCP
connection is forwarded through SSH to the remote port 5000 and
handled by
socat
. When receiving the connection,
socat
spawns the
rsync --server
command we got at the previous step and connects its
standard input and output to the incoming TCP socket.
$ ssh -L 127.0.0.1:5000:127.0.0.1:5000 web05 $ socat tcp-listen:5000,reuseaddr exec:"rsync --server -vlogDtpre.iLsfxCIvu . /data/."
The last step is to connect to the source with
remote
port
forwarding.
socat
is used in place of a regular SSH connection and
connects its standard input and output to a TCP socket connected to the
remote port 5000. Thanks to the remote port forwarding, SSH forwards
the data to the local port 5000. From there, it is relayed back to the
destination, as described in the previous step.
$ ssh -R 127.0.0.1:5000:127.0.0.1:5000 web04 $ rsync -e 'sh -c "socat stdio tcp-connect:127.0.0.1:5000"' -aLv /data/. remote:/data/. sending incremental file listhaproxy.debian.net/haproxy.debian.net/dists/buster-backports-1.8/Contents-amd64.bz2haproxy.debian.net/dists/buster-backports-1.8/Contents-i386.bz2[…]media.luffy.cx/videos/2021-frnog34-jerikan/progressive.mp4sent 921,719,453 bytes received 26,939 bytes 7,229,383.47 bytes/sectotal size is 7,526,872,300 speedup is 8.17
This little diagram may help understand how everything fits together:
The
rsync manual page
prohibits the use of
--server
. Use this
hack at your own risk!
The options
--server
and--sender
are used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these options may be needed in certain scenarios, such as when setting up a login that can only run an rsync command. For instance, the support directory of the rsync distribution has an example script namedrrsync
(for restricted rsync) that can be used with a restricted ssh login.
Addendum
I was hoping to get something similar with a one-liner. But this does not work!
$ socat \> exec:"ssh web04 rsync --server --sender -vlLogDtpre.iLsfxCIvu . /data/."\> exec:"ssh web05 rsync --server -vlogDtpre.iLsfxCIvu /data/. /data/."\over-long vstring received (511 > 255)over-long vstring received (511 > 255)rsync error: requested action not supported (code 4) at compat.c(387) [sender=3.2.3]rsync error: requested action not supported (code 4) at compat.c(387) [Receiver=3.2.3]socat[878291] E waitpid(): child 878292 exited with status 4socat[878291] E waitpid(): child 878293 exited with status 4
And SSH agent forwarding is dangerous . Don’t use it if you can. ↩︎
Značky: #Debian