Hacker News Reader

Comment by tyingq | Hacker News Reader

Tar combined with ssh is a decent replacement for scp. A little more dangerous if you're in the wrong directory, etc, but...

  $ cd /source/dir && tar -cf - . | ssh rhost 'cd /dest/dir && tar -xvf -'

Similar for the other direction.

View on HN

I prefer to use rsync over SSH as a replacement for scp:

alias scp='/usr/bin/rsync --archive --xattrs --acls --progress --rsh="ssh"'

I have completely deprecated for myself the use of both scp and sftp many years ago and I disable the sftp server (in sshd.conf) on all my servers.

While scp and sftp are also slower, usually being unable to reach link speed on fast links, the main reason is that I have discovered that when copying files with them they sometimes were losing file metadata (e.g. parts of the timestamps or extended attributes) without any warnings or errors.

rsync does not have such problems, it can make exact copies even when copying between different operating systems or different file systems.

Because I have stopped using both scp and sftp many years ago, I do not know whether meanwhile there was any effort to remove the bugs from scp and sftp, but I doubt it.

I agree rsync is better than using tar, but often I'm using that ssh/tar command line before rsync is available on the other end.

> before rsync is available on the other end.

I’m curious, is this in an embedded environment or interacting with some sort of boot loader?

Nothing so drastic; I often find that Debian/Ubuntu VPS servers do not come with rsync preinstalled.

And rsync can be curiously slow under some circumstances.

Tar can also be used with, e.g., netcat (included in busybox), useful for bootstrapping / building / rescuing systems.

why the explicit rsh?

The rsync option --rsh="command_name" or -e "command_name" is required to be certain that "ssh" is used for the transfers.

The rsync program might have been compiled to use "ssh" even if you do not use the --rsh= or -e options, but you cannot know this for sure (unless you have compiled it yourself and you have read the sources to verify that).

I suggest also adding -H (hard links) and -S (sparse files) to that command-line.

Not sure if it's common across all tar implementations, but you can use -C to let GNU tar change directories before performing any operation. So this command is equivalent:

  $ tar -cC /source/dir . | ssh rhost tar -xvC /dest/dir

(I also left out -f - because it's the default)

Helpful, thanks. Old habits die hard. Tar wasn't always GNU tar back when this became useful to me.

FWIW, while tar isn't even a POSIX command, -C is supported by bsdtar[1] (including macOS), AIX tar (confirmed AIX 7.1), and Solaris tar (confirmed Solaris 11.4). Argument ordering and bundling is really funky on AIX and Solaris, though, especially for extraction.

[1] Seems like everybody except OpenBSD has migrated to libarchive's bsdtar, but OpenBSD also supports -C.

Just a sidenote, this is probably the fastest way to copy lots of tiny files across a high latency link (but without -v [1]). Most protocols transfer files individually, waiting for the server to finish writing it - adding a tiny delay for each file transferred. The tar above simply pushes the files across the pipe as fast as it can without waiting, while the receiving tar can write them as soon as the data is received. Recommended to pair with lzop for fast compression.

Second sidenote, using this with pv is a great way to view total progress. http://www.ivarch.com/programs/pv.shtml

1. With -v every filename is written to the terminal, which causes context switches and IO waits. This can significantly slow file copies with lots of tiny files.

Reply on news.ycombinator.com