Hacker News Reader

Comment by yuvadam | Hacker News Reader

Current status: bad things are happening in the North Virginia datacenter.

EC2, EBS and RDS are all down on US-east-1.

Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

Indeed they are. Right before the issues began, I pushed a bad update to one of my Heroku apps, causing it to crash. A minute later I fixed the bug, re-pushed the git repo to Heroku... and nothing. I've been stuck with an error message on my website for hours. Unfortunate timing!

Staging servers are an easy thing on Heroku :)

You have my sincere sympathy.

http://status.heroku.com/ for those on Heroku

Is it just me, or is their status page down?

It's been on-and-off for the past few hours.. It's up right now (for me, anyway).

This morning from 5a - 6a Pacific time I was able to access my Heroku app just fine.

I can access all my heroku apps that have their own DNS. Anything with a .heroku.com subdomain is down for me. Frustrating, knowing that the apps are still running but aren't routable.

Yay, cloud.

Please do not make content-free posts such as this. It adds no value to the conversation and is only noise.

If you actually wish to make a useful point about the practicality or otherwise of massively virtualised systems for webapp deployment, please do. It's going to take more than two words though.

Please do not make content-free posts such as this. It adds no value to the conversation and is only noise.

If you actually wish to bitch about a post that we all got the point of, you're going to need more than two paragraphs.

I guess you haven't seen the Microsoft ads about the cloud? http://www.youtube.com/watch?v=Lel3swo4RMc

Anyway, my bad, I was just trying to make a joke to lighten up the mood. Sorry.

You're right, I hadn't. Fair enough.

I laughed, it's relevant and puts things in perspective if you had seen the ads. So it's not really content free even if it's just two words.

I know you probably have your answer by now, but this site always helps me out when I have that question:

http://www.downforeveryoneorjustme.com

Foursquare is up now. Quora not showing the 503 anymore, Reddit still down.

I stand corrected. 4sq down again.

We rely heavily on EBS still, so this is hurting us more than most others. Hopefully they'll have us back up soon.

You guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?

We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option.

So I guess that is the long way of saying that hopefully it won't happen again.

How do you move away from EBS and still deal with large data?

I imagine they'd do consider some combination of the following (sorted by most likely)

1. Sharding data 2. Pulling tables out to other servers from the main DB 3. Pruning excessive data 4. Compressing data

It still has to be stored somewhere though right? If it's EBS you've just made yourself a complicated solution that will eventually fail all over again. No?

If the data is sharded, then the data/server is small enough enough to fit within the individual server's disk and you no longer need EBS to store it.

Not sure what you had in mind by "large", but instance storage goes up to 1.7TB: http://aws.amazon.com/ec2/instance-types/

The reason Reddit uses RAID10 is for performance, not disk size. A single instance storage device is just too slow for the Reddit database.

Many instance types have 2 or 4 virtual disks (presumably on different physical disks).

I do not believe you could be effective by moving away from EBS, you know without giving up quite a bit.

Doing things the right way with EC2 means using EBS. It's the brake caliper to the rotor. Sure you could have drum brakes but they're not nearly anywhere effective as they quickly get heat soaked. I'm referring to S3.

One should trust ephemeral storage. Your instance can go down at anytime. Write speeds to S3 are not nearly as fast as ephemeral or EBS arrays (raid).

Hate to say it, but If one cannot trust EBS then what the heck are 'we' doing on EC2... EBS quality should be priority one, otherwise we're all building Skyscrapers on foam foundations of candy cane rebar.

I can't say whether much has changed within the last year, but when I worked at FathomDB we had serious issues with EBS. You couldn't trust it. Odd things would happen like disks getting stuck in a reattaching state for days and disks having poor performance.

Please tell us how you plan on moving 700 EBS volumes to something completely different. It sounds amazing.

NOTE: I work for Gluster.

We have had a lot of success stabilizing EBS by creating mdadm arrays out of lots of smaller EBS volumes. There is minimal additional costs and you can get better performance, stability, and protection (RAID 5, 6).

Gluster makes an OSS distributed filesystem that runs across availability zones, our AMI (not OSS) builds multiple RAID arrays on each instance then spreads the filesystem across instances in multiple AZs. Send me an email if you want to chat.

Foursquare is up, but Quora is still showing the 503.

Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.

Same with us. About 10% of our 700+ volumes are having problems right now.

It's hard to tell for sure since there isn't any load.

We're fine on EC2 -- but everything on RDS seems to be giving us big problems. We started a few backups before we new it was systemic, and all of them are stuck at 0%. We also tried spinning up new instances -- and they're all stuck in booting.

We had about 45 min of downtime around 4am EST. Our RDS instances, EBS backed and normal instances all returned without problems. We are in Virginia us-east-1a and us-east-1b.

I think which physical data center "us-east-1a" etc. corresponds to differs from user to user, to load-balance given that people will probably be more likely to use 1a than the other zones.

Also Cuorizini (http://cuorizini.heroku.com) is down!

Reply on news.ycombinator.com