Hacker News Reader

That is one (good) way to handle the encryption.

You don't want to leave out the compression or block size however, so add those back in. Most WAN links are low bandwidth enough that compression will not slow things down (this is usually true even on 1 GBE LAN links for pigz) and in my experience the speed-up is substantial.

pigz is much faster than using ssh compression as it is multicore. apt-get it or http://zlib.net/pigz/

View on HN

the compression is unnecessary here, you're moving a max of less than a couple of hundred GB of data.

A few hundred GB here, a few hundred GB there, pretty soon you're talking real bandwidth.

It's pretty necessary if you aren't moving between colocated boxes. "less than a couple hundred GB of data" is still a lot of data to be moving around, even on a 100Mb/s link, which almost no one in America has residentially, and isn't even a guarantee for colo'd machines.

In most cases, compression at the raw block level will result in HUGE size savings, especially since a lot of that may be free space. It might even make this transfer tenable.

Just add -C to the ssh command to enable compression.

Reply on news.ycombinator.com