Amazon EC2 EU-West is down | Hacker News Reader

Amazon EC2 EU-West is down

109 pointsdabeeeenster31 comments

Nothing yet on the AWS service status but Twitter is going a bit crazy.

UPDATE 19.15 BST: now being reported as network issues by Amazon

UPDATE 19.24 BST: Can now see my ec2 instances again from the management console but can't ssh into most of them.

UPDATE 19.27 BST: From Amazon: EC2 APIs in the EU-WEST-1 region are currently impaired. We are working to restore full service. We are also investigating instance connectivity that we believe to be limited to a single Availability Zone.

UPDATE 19.34 BST: RDS and Route 53 also affected.

UPDATE 20.10 BST: From Amazon: The issues with the affected Availability Zone are the result of a power failure in that zone. We are currently recovering power and anticipate that instances in the effected available zone will start to recover within the next 30-60 minutes.

Well, that sucks.

View on HN

I run a site that monitors over 100 different cloud services. Our monitoring node in EC2 EU West went down at 10:48:51 PDT. Our monitored application with PHP PaaS cloudControl, also hosted in EC2 EU West, went down at the same time. For the past year, we've gotten 99.978% availability in this EC2 region: http://cloudharmony.com/status

Power outage?

Interesting -- Linode also went down today because of power outage: http://status.linode.com/2011/08/outage-in-fremont-facility....

Is there any chance these power failures have something to do with the solar storms we are experiencing right now?

They have subsided. Note the K-index on this page: http://spaceweather.com

This is what, the 3rd? 4th? power outage in that facility in the last year? I realize that it's not directly Linode's fault, since this is Hurricane Electric's facility that they lease from, but its still pretty frustrating. I have a personal VPS there which I use to do an hourly data grab, so I missed a few this morning. This is just a personal pet project so no huge deal, but if I was running some actual production-level stuff there I would be very wary of using that facility.

Linode doesn't notify their customers affected by outages, or am I missing something?

I've been notified for what amounted to two minutes of downtime before (my VPS is in the London data centre). Are you in the Fremont data centre? That is where this outage was.

Yes, I have a machine in Fremont. How were you notified? Email?

Yes. An email about unexpected downtime, then another two minutes later saying the downtime was over.

Hmm, I don't see any way in my Linode account settings to disable these emails, and I never received any! That is disconcerting. :(

They don't. They do have an RSS feed that you can monitor with your favourite feed aggregator: http://status.linode.com/rss.xml

Thanks for the link.

So that's what interrupted my screen session.

A must-follow Twitter feed for anyone using Amazon AWS is @ylastic: http://twitter.com/#!/ylastic

At the very least, even if you don't use Twitter, it's a good resource to check for up-to-the-minute info if you're seeing an issue and want to verify.

THANK YOU HN!!! I couldn't connect to my machine and couldn't understand what was going on... good thing I turn to HN as soon as I get online...

You'd think AWS could afford to send emails to keep us customers up to date with the problem?

If they sent you an email every time there was downtime, then you would know every time there was downtime. Any time your servers are not available but you don't notice == win.

> Any time your servers are not available but you don't notice

Can't happen. My servers are monitored by external services that (to the best of my knowledge) don't use AWS.

It's best to play the open card in any case.

Emails will be sent out when the problem is more fully handled, as usual. HN is a decent place to get some info, but you should consider posting to the AWS developer forum as well, b/c there are Amazon people constantly checking on things and answering questions:

https://forums.aws.amazon.com/forum.jspa?forumID=30

Edit: I should add, they can look up your specific instances to get the exact problem.

Looks like a lightning strike in Dublin knocked out power at data hubs for Amazon and Microsoft: http://news.ycombinator.com/item?id=2858239

The specific availability zone they're talking about is eu-west-1a. Servers we have there are reported as 'running', but are unresponsive.

Other zones seem fine, but starting new servers doesn't seem to be working.

Availability zones are different for different accounts, so for me it's eu-west-1b. You can log into the console and under the EC2 tab it will show you the status of each of your availability zones.

Ah, good to know, thanks for the pointer.

Another update on the status page, btw. It appears to be a power failure.

I've tried to create a machine in another zone with no success. I suppose there's a lot of people doing the same :(

If anyone is interested, ServerDensity thinks that our machine went down between 6:37 and 6:42 BST.

As an AWS customer affected by this, that's the time our servers went dark.

I can SSH to some of my newer instances but not the older ones.

I see all my servers on the management console, but I'm unable to ssh any of them, all in the same availability zone, eu-west-1b

Has anyone gotten any timeframe when the rest of the RDS is going to start responding? Not much info on the status page of AWS.

I can see my server from the management console but I can't ssh to it. Damn, my IRC is down :(

CloudFormation is also affected now.

Clear skies over Dublin, not a cloud in sight. ;)

Reply on news.ycombinator.com