Hacker News Reader

Comment by voidlogic | Hacker News Reader

If your using lvm you can create a snapshot to do this while online, if not just read from your disk (assuming sda here):

   1. (offline) Boot new and old VM servers from live CD
   2. old server: dd if=/dev/sda bs=8M | pbzip2 -c | netcat <newhost> <random high port>
   3. new server: netcat -l <same port> | pbzip2 -cd | dd of=/dev/sda bs=8M

Compression: You can use something besides pbzip2, maybe pigz of if you only have a single core use bzip2 or gzip. Security: You probably want to add encryption to this pipeline.

View on HN

Woah woah woah isn't this just transferring the contents of /dev/sda in the clear over the wire? Shouldn't you at least do this over SSH?

>>Shouldn't you at least do this over SSH?

Yeah, I mentioned that at the bottom of my post. Using ssh or some other inline encryption would be a good idea if it is a system you care about. If you have a site to site VPN tunnel between your systems, you can skip adding the encryption.

Yeah, I mentioned that at the bottom of my post

It wasn't there when windsurfer replied to you (I was reading the thread earlier), hence his question.

Thanks. I thought I was just dumb.

I thought my only edit had been to replace might with probably. If this was not the case I'm sorry. Maybe I edited it to add that right after posting and forgot-

dd if=/dev/sda bs=1M | ssh root@<newhost> dd of=/dev/sda bs=1M

let SSH handle compression for you instead.

That is one (good) way to handle the encryption.

You don't want to leave out the compression or block size however, so add those back in. Most WAN links are low bandwidth enough that compression will not slow things down (this is usually true even on 1 GBE LAN links for pigz) and in my experience the speed-up is substantial.

pigz is much faster than using ssh compression as it is multicore. apt-get it or http://zlib.net/pigz/

the compression is unnecessary here, you're moving a max of less than a couple of hundred GB of data.

A few hundred GB here, a few hundred GB there, pretty soon you're talking real bandwidth.

It's pretty necessary if you aren't moving between colocated boxes. "less than a couple hundred GB of data" is still a lot of data to be moving around, even on a 100Mb/s link, which almost no one in America has residentially, and isn't even a guarantee for colo'd machines.

In most cases, compression at the raw block level will result in HUGE size savings, especially since a lot of that may be free space. It might even make this transfer tenable.

Just add -C to the ssh command to enable compression.

That's a great way to screw up the filesystem on the destination. You'll get mounted fs write combined with low-level block writes, filesystem in a blender.

LVM snapshot, that's the way I'd go about it. Is how we do our backups, to maintain a good state. Mount the snapshot, and do the same with it.

There's very nice utility called Ghost 4 Unix that makes the whole thing even easier: http://www.feyrer.de/g4u/

Looks like overkill for a simple one time task that can easily be handled the UNIX way. Worth looking into if you need to do this alot-

Your procedure already requires booting out of live cd, and then uses "unix way" that might be easy to screw up for less advanced people, so using G4U simplifies it substantially, does the same good job and by no means is a 'overkill' ;)

DigitalOcean should probably put this in their marketing materials.

Reply on news.ycombinator.com