Hacker News Reader

Comment by IgorPartola | Hacker News Reader

> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime.

So what do you use now for your persistent storage? This might be the most interesting part.

View on HN

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about?

PS great article

As others have pointed out, all EC2 instances come with large ephemeral volumes already attached as /dev/sdb, etc; you just have to mount them:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/In...

Even the smallest instances have 160GB of storage, all the way up to 3TB on the cc2.8xlarge. New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage. (just tested it last week)

New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage.

Just a tongue-in-cheek comment: Those instances also cost over $2000 USD per month. For the same money you could buy a new Dell Server with 2-4 SSDs, every month. ;)

Lol, very true, but for 3-4 hour workloads they are perfect- which I think is one of the "makes the most sense" kind of EC2 workloads.

In this case, I had about 8 million output files from a Hadoop job that I needed to process with MySQL- it was perfect!

THey're only $2000 if you don't have reserved instances.

Each EC2 instance type has a different amount of ephemeral storage, from 160 GB to 3.3 TB. You need to arrange for it to be mounted when you launch the instance.

a different amount of ephemeral storage, from 160 GB to 3.3 TB

Except for t1 and m3, which have zero ephemeral storage.

This is a good approach if you can get enough redundancy with your instances and if instance failure is uncorrelated.

However, what do you do if there's a massive outage that affects most or all of your instances simultaneously? Diversification that fails when you need it the most isn't very helpful.

As I mention in the post, we go across AWS availability zones: these systems have physically separate power and networking. When they fail together, it's because they are connected at the software level -- usually EBS. If you're not using EBS, then Amazon's multi-AZ pattern is sufficient: just make sure your instances are spread across AZs.

Of course, east-1 has fallen off the map entirely on at least one occasion -- for that reason we keep a "seed" set of warm databases in us-west; if east-1 were to have an extended outage we have a disaster recovery plan that involves transferring DNS to IPs in west and spinning up new app instances there.

Reply on news.ycombinator.com