Hurricane Sandy and AWS: Migrating Octopart Out of us-east-1 in a Hurry | Hacker News Reader

Hurricane Sandy and AWS: Migrating Octopart Out of us-east-1 in a Hurry

octopart.com

53 pointssam22 comments

Ok, if you're looking for a startup idea, here you go a free one on the house :-)

"EjectorSeat" - EjectorSeat is the hip cool way to create a current, usable, set of scripts which can move any AWS/EC2/Linode/Heroku/... (start with AWS/EC2) instance from where it is, to somewhere else. By using EjectorSeat all of the changes and specs you have on your install are automatically documented into its configuration and data location databases so that when the time comes to pull the yellow and black handles, you will know that your install is being migrated, and better yet, when its done migrating one push of a button and blam! its up and running.

-----------

It seems like this should be possible to package up (it will take a bit of work of course) but its the 'copper' option (of the silver/bronze/gold nomenclature) for disaster preparedness. Basically you don't have the funds to maintain two instances all the time, this lets you move the one you have when you need to.

Of course like real ejector seats it will have failure conditions (like flying inverted at 50', bad time to eject) but it could provide an ops guy with a bit of piece of mind.

Most of the hard work is already done for this: Puppet, Chef, etc.

CloudFormation should also be helpful here.

Yeah - but what would really help is if EC2 had more tools to support cross region moves†.

Hint, hint…

† I too did an us-east-1 to us-west-2 move on Monday, and it was unfun.

They provide pretty good tools already. If you leverage their API and S3 you can migrate AMI's and EBS's across regions quite effectively. Could the tools get better, yes. But it's not as hard as it is often made out to be.

I just did this Monday, and there is 0 support that I could find for moving ebs volumes cross region. I used netcat to push the data, which seems profoundly wasteful.

I'd love to hear of an s3 based approach.

You can use this, many people have reported it does well. I have not used it.

https://cloudyscripts.com/tool/show/5

You can also use the APIs to do that S3 copy I mentioned. Here are one of the many write ups of that technique.

http://alestic.com/2010/10/ec2-ami-copy

I wonder what percentage of AWS clients are not using Puppet, Chef or similar. I bet more than half.

If I were going to build this, I'd take a look at Blueprint, since it (at least in theory) can reverse engineer what's installed on a running server.

Agreed. We created a backup database in a west region and it only took a few minutes to provision with Chef.

I would be curious if there was an open source tool that copies your current security groups and writes them to other regions.

LOL on Ejection Seat reference. Can we add NATOPS procedures for emergency APU start in flight? That would maintain power to systems in case of double flame-out.

I had a similar idea for a while. The memories of trying to get EC2 nodes manually configured, and it isn't easy at first. Also some painful memories migrating clustered servers (Linux running H-Sphere Control Panel) which didn't exactly fail gracefully.

Even with puppet/chef and other tools, migration takes time and planning. EC2 node architecture is not persistent, and should factor in failure of a node at any time.

Linode makes it easy to move instances around to different regions - at the hypervisor level it's just a command to move a Xen instance within their network of DCs.

Moving between vendors is always going to require more testing/configuring/etc then just a script to move the assets over.

RDS/EBS snapshots being bound to their region is a big stumbling block. Especially for RDS, because the only solution is mysqldump (or similar) and that not only blocks the production database but takes hours for databases above 10 GB. So at the end of this you have a very out-of-date backup that you can transfer to the other region.

Cross-region snapshot mirroring would be a great enhancement feature for AWS.

You might just consider setting up a dedicated slave MySQL database on a smaller instance in the west region just for the eventuality that you need to spin up a master there. Then you can just snapshot, copy to ephemeral, and go from there.

Given that the alternative is being down for days, turning a site read-only while the dump-move is happening seems like a reasonable compromise.

"EBS snapshots cannot be moved between regions"

Not true. I've done this using Ylastic. See http://ylastic.com/features.html:

"Migrate EBS linux snapshots between regions."

"Migrate EBS windows snapshots between regions."

Takes a few minutes to kick off and does it all for you.

Interesting. I wonder if they use an undocumented AWS feature or are snapshotting the volumes and copying them. My guess is the latter.

The OA comment of "EBS snapshots cannot be moved between regions" should have "easily" inserted in there. I've seen quite a few full length processes for doing it, e.g:

http://elastic-security.com/2011/02/10/how-to-copy-an-ebs-ba...

http://serverfault.com/questions/336321/how-do-i-migrate-ama...

I'm 95% sure they are just putting a UI over that lengthy process. But as I say, I've tried it and it worked perfectly for me. This is their process:

http://blog.ylastic.com/migrating-snapshots-between-ec2-regi...

As you’ve used it - how long does it take? I read up, and I decided they where probably just making a new EBS volume in the new region, `netcat`ing the data across, and taking a snapshot of that new volume, with the `netcat`ed data.

So I did it by hand - that way. It it’d be pretty easy to tell if this was the case, because it really wouldn’t be fast….

It's not fast, they send you an email when it's done. It was a while ago now, but I recall it taking about 30 minutes before I got the email.

I know this is a little late for your case but you can migrate snapshots from 1 region to another. We spend sunday moving from US-East 1 to US-West 1 and took 2 hours to setup our entire infrastructure consisting of 20 servers & about 180 GB database

There are several options

1. Manually - http://alestic.com/2010/10/ec2-ami-copy

2. Automatically via Scripts a) migrate-ebs-image.pl - http://search.cpan.org/~lds/VM-EC2/bin/migrate-ebs-image.pl b) CloudyScript Migrate SnapShot - https://cloudyscripts.com/tool/show/4 c) CloudyScript Migrate AMI - https://cloudyscripts.com/tool/show/5

3. Commercial Services a) http://ylastic.com/ b) RightScale

* ylastic is really cheap at $25 / month * I used the perl script migrate-ebs-image.pl, super simple to install and use. * CloudyScripts has open sourced their Ruby gem so you can build on top of it. They also have a free web form you can just use (but may not be secure enough). But you can launch their AMI in your own instance which should be secure

Hope that helps for next time!

Thanks, these will be useful for next time. Hopefully AWS will add the capability to migrate the EBS snapshots to the console soon.

would be nice for sure. luckily AWS builds the API first then builds it into the console, so many you can still get the job done just takes a little digging. feel free to reach out if you ever run into another situation like this

Reply on news.ycombinator.com