Hacker News Reader

Comment by mjschultz | Hacker News Reader

Along the same lines as number 1. You can transfer a directory with

  $ ssh user@host "tar cvzf - /path" | tar xvzf - /path

View on HN

This is especially useful when dealing with directories with too many files in them - say around a million files. Regular globbing won't work, and most regular tools won't work - but tar works great.

Also - another trick is to give it -c blowfish to use a lighter/faster cipher for the transfer to save on CPU time.

Ssh already compresses, so you can drop the z. And generally cpio does a better job preserving the file system structure (hard links, time stamps). And it tends to be a bit more cross-platform, if you don't have GNU tar on both ends.

ssh(1) only compresses if configured too, see the -C option.

You can also just transfer it with:

$ scp -r user@host:/path /path

I usually do use scp, but it becomes problematic when there are symlinks hanging around. scp just follows them and ends up copying the same file many times.

This is especially bad if you have a directory structure that references a higher-level directory.

Use rsync then. It can copy symlinks and even recreate hardlinks.

In my experience, using rsync to do just a raw copy is not faster than the ssh+tar approach. rsync wants to poke around and optimize, and that's incredibly awesome, except when there is absolutely no chance of optimizing at all, in which case it's just dead weight. I don't know of anything that can beat ssh+tar (possibly with the encryption tuned cheaper, as mentioned above); it especially blows raw scp out of the water for lots of small files.

scp is a lot slower than the tar-pipe with many small files.

Reply on news.ycombinator.com