Network file transfer with on-the-fly compression

codelathe.com

9 pointscodemechanic6 comments

I'd like to see a comparison between this and scp with compression.

That's the first thing I thought of too. The -C flag is there specifically for this very scenario.

Also, would "j" have worked better than "z" in the tar line? bzip2 is much slower than gzip, but does a better job.

Really we should have adaptive compression so that the CPU and network usage is balanced. Too bad it may require "rampant layering violation" to implement.

scp -C does the same thing. Both methods use gzip for compression.

But the tar method will generally be 5-15x faster for large collections of small files.

Handing it with tar makes operations on both ends asynchronous. scp (or rsync, etc.) of multiple files does file opens on each end in lockstep, resulting in a lot of idle wait time.

Note that this is not likely to help with media files (mp3, mpeg, jpeg).

Reply on news.ycombinator.com