Global Outage of AWS CloudFront CDN on Nov 26 2014 | Hacker News Reader

Global Outage of AWS CloudFront CDN on Nov 26 2014

turbobytes.com

89 pointssajal8336 comments

It's the second time in less than a week that AWS have failed to update their status page during service disruption and then filed it under 'increased error rates'.

Don't get me wrong, I love AWS and use it every day, but it's a little insulting to pretend that things haven't gone wrong when they quite obviously have.

I know Amazon aren't the most transparent company in the world but why even bother having a status page if everyone has to turn to twitter to validate the outage?

Last night while sitting around waiting for cloudfront to come back so I could get on with my work I turned to HN to give me a bit of reading material. My word, sooooo many of the articles linked to on the front page were missing their static assets because of the AWS outage. Would be really interesting to know what percentage of the internet was affected.

1&1 are guilty of this too.

1&1 is the scum of the internet.

care to explain this ? whats wrong with 1&1 ?

Bad services. Someone even social engineered our 1&1 account a few years back...

Terrible technical setup, we have a dedicated server that can't be guaranteed an upgrade without losing the absolute server paths.

Its not the first time. Their status page is not trustworthy. When there was an issue a few weeks ago concerning the HVM? many instances were restarted. Status page showed green as if nothing had happened. They probably change it when its well known.

Its conflicting for a company to show its own status page I suppose. Its in its own interest to get it to be green as much as possible, even if its just for display's sake as a false sense of security.

Sounds like it's time for one of us to make an AWS status page. I wonder if it'd be possible?

Seconded

At least for EC2 status updates and their color (green i, etc) are only posted if they meet certain thresholds. IE: Greater than $X number of instances are actually offline or unavailable etc.

Possibly many of the outages you happen to be seeing are in reality not that large thus not meeting the threshold to have a post, or major status update.

Don't host it on AWS. :)

This is actually a hard problem because many of these services are localized using BGP anycasting. So the status checking would have to have probe nodes all over the world.

We learned about this particular outage from health checks we had set up in Pingdom. Somebody like them could expose those checks as a public service, and try to become the go-to place for getting the real status of AWS.

Are there independent services that monitor a range of hosting providers, cdn, etc. and keep a history that can be used for reference? I'm thinking someone like pingdom could do this.

https://cloudharmony.com/ does to some extent, though it's not exactly what your talking about

check out cedexis http://www.cedexis.com/reports/#?report=cdn_response_time&co...

Grammatical aside: I've noticed more usage of "company x are…" I never really went for "the data are/show/etc.", despite the learned Latin explanation. To me, "the data" indicates the same group as "the information" and is still singular. A company acting as an entity is singular. I hope the "company are" trend in writing doesn't catch on. (I never much liked "graphical" either) Flame away. :)

In American English, the correct form is "Company X is" where the company is considered singular. However, in British English, the correct form is/can be "Company X are" where the company is considered plural.

http://english.stackexchange.com/questions/1338/are-collecti...

We started using Google Cloud Storage when it first came out so that user uploads would be closer to them (in theory) due to GCS' distributed design. They seem to have periods of 15-90 minutes every few months where they start timing out or giving us huge errors and often times we don't even hear about it either. They don't have a status page any only announce via email list - usually way after we've been scrambling.

Anybody else have a bad experience with Google Cloud Storage? I'm wondering if we're better off going with S3. Most users on my site are uploading photos, so when that dies, everything is down.

The Google Cloud status page is at https://status.cloud.google.com/

And there's a history page too, listing two outages for Cloud Storage:

https://status.cloud.google.com/summary

I'm really freaking tired of the AWS Status page using the green "info" marker for major outages.

bottom line: always CNAME your asset domain.

Totally. We were able to side step the CDN in less than a minute. +1

That helps! I guess it assumes your origin servers have the capacity to serve the traffic. Or did you switch CDNs? Just curious as there are services like Cedexis (or TurboBytes; the author of the article) for load balancing CDNs

What exactly is an asset domain? The one you setup to point to the CDN?

Yes. Instead of using pqowieurpqoiwer.cloudfront.net, you can point one of your own subdomains at it. Leaves you in control. The only complexity is SSL certificates - you either have to pay $600/month for a custom cert, or use SNI and ditch some older IEs.

$600 per month for what exactly? (honestly asking here)

Without SNI, you need a dedicated IP per certificate. For CloudFront, this means dozens of dedicated IPs around the world just for your distribution.

Ah, makes sense! Thanks

Could you add a global variable to your site code with the fqdn of your assets to point to the origin for times like this?

They do offer free SSL with SNI. But that's not viable for most people given the number of popular older browsers missing SNI support.

Browsers that do support SNI include:

IE7+ (unless running on Windows XP)

Firefox 2.0 and later

Opera 8.0 and later

Chrome 6 and later (unless running on Windows XP)

Safari 3.0 and later

Android default browser on Android 3.0 and later (this is probably the biggest chunk of users)

Windows Phone 7

So, realistically, you're looking at people who still use Windows XP (unless they're using Firefox) and people with really old Android phones that'll never receive a manufacturer firmware update.

Meh. Adds an extra DNS lookup.

Just have a short TTL and let people who ignore it suffer. ;-)

And burn down your backend servers. Assuming you are not swithing to another CDN that fast.

This is incredibly upsetting. Amazon publishing false status reports for public consumption is directly equivalent to libeling every single customer they have, simultaneously.

I was trying to register a new domain during this outage. When my registrar's site wasn't loading correctly, I dug around and discovered they use AWS. I then checked Amazon's AWS status page to see if they were the culprit.

Nothing but green icons across the board.

Amazon having the audacity to tell such a gargantuan lie seemed inconceivable, so I was absolutely convinced that the problem was the registrar's fault. I decided the registrar must have become shoddy at some point without me noticing, and so I made a "to-do" note for myself to switch to another registrar this weekend.

If I hadn't happened to stumble across this HN submission, the registrar I use would have lost a long-time customer due to Amazon publishing a maliciously false status page for AWS.

Is there some way to do a fallback if the DNS server is down?

Also if my host is down can I "quickly" try another host? I know that it's possible in my own app, but can I tell a browser to do it, as a web publisher?

Reply on news.ycombinator.com