Hacker News Reader

Comment by seldo | Hacker News Reader

Waking up at 2.30am to a total service outage sucks, but at least I know all you bastards are awake and screaming too. That makes me feel slightly better.

(We're back now. Kudos to Pingdom for noticing and alerting)

View on HN

Ah, pingdom failed to alert us via push got only the email alert (luckily it was exactly when the team is warmed up and ready to hit the keyboard :) 11AM at Israel)

I have an IFTTT recipe to get Push notification if you have a server outage http://ifttt.com/recipes/25122

Cool! thanks, adding it now (IFTT rocks)

For a good 20 minutes I didn't know it was an outage and panicked thinking my server mysteriously stopped working, and couldn't even SSH in.

I've learned that when my SSH session dies, the first thing to do is go to twitter and search for "EC2". People were complaining there within about 60 seconds of the outage starting.

Alternative monitoring opportunity? Grep a twitter stream for "EC2" and "down" and raise an alert if you get enough hits in 5 min?

Looks like the entire US-East-1 region had connectivity issues. We're a monitoring company, we're watching this very carefully...there is a write up on our blog:

http://www.verelo.com/blog/2012/03/15/aws-entire-an-region-d...

What's funny is that this is what I find myself doing instinctually when I encounter an outage or high latencies on ANY service or site. Heroku (recent process startup woes), Google Apps (slowdowns, specifically Gmail), Amazon (when its hammered by traffic to big deals), etc etc.

I second the comment above suggesting a "crowdsourced" status app monitoring twitter. Although it's no consolation for service interruptions, it does at least keep you sane knowing the problem is elsewhere.

Yes, because Twitter never goes down.

What does it matter if Twitter goes down? The odds of it happening for an extended time right at the beginning of an EC2 outage are rather small, and even in that worst-case scenario, it doesn't really put us in any worse of a position than we're in now.

Touché.

Reply on news.ycombinator.com