Hacker News Reader

Hacker News

Comment by akulbe | Hacker News Reader

Full thread

akulbe

Maybe I'm crazy, but it seems too coincidental that multiple services

(Amazon, Facebook, Apple... that I know of, so far)

have been affected, on the same day.

View on HN

It's related to Akamai DNS.

Do you have a source?

Trying to access some APIs I work with (DEP,V PP) earlier today and seeing 'akadns' as the common providrer.

Charter (ISP) was also out for over an hour—first time this year it was a complete outage and not just a DNS issue.

I've been on Charter all day VPN'd into work with no interruption. Although I have a business connection at home, so it's possible I'm avoiding whatever middleboxes they jam residential traffic into.

my ISP Cox, had server failure for several hours about 2 weeks ago. It was a DNS issue as well. I don't want to take 2 unrelated things and call them similar, but one has to imagine if some of the above posters are substantiated several huge players losing peering/connectivity and 2 very large ISPs having non-trivial multi-hour multi-region DNS outages is bad. Horrible coincidence.

I have been losing google docs intermittenly all day (and I have been working nearly exclusively in google docs all day) maybe just me, but I have been unable to upload photos (same one as yesterday to gdocs) and from drive/docs dash I would lose connectivity and have sever lag loading up a doc I just worked in. Had to close browser several times.

Maybe one of the backbone carriers has a bad day?

Perhaps, but I (network engineer at an ISP) haven't heard anything like that, and all of these services have multiple upstreams and private peerings.

Comcast has as few peerings as they can legally get away with. Redundancy isn't high on their list of things to care about, apparently.

> ... as they can legally get away with.

Can you clarify a bit? I'm not sure what you mean.

An ISP isn't, by law, required to peer with anyone.

If they didn't peer with anyone they'd be a closed network and it would be illegal to advertise that as "internet access".

They're legally required to live up to the claims of their product. If they say X Mbit/s then there needs to be at least a plausible case made that they can achieve those results. Anything significantly less is to invite a class-action lawsuit.

Well that was strange. Wonder what it was.

Why?

New NSA program rolling out? O_O

No. That was the Syria/Iraq/Iran BGP outages from earlier today.

You laugh, but I guarantee you that the NSA has systems reliability issues just like the rest of us.

The last person that guaranteed me something about the NSA is now exiled in Russia ;)

This comment turned out to be surprisingly controversial for a pretty obvious joke. I've observed it at +4, down to +1, back up to +3, and now it's at 0! Heh. Didn't mean to rustle jimmies of any NSA contractors that may read HN. I'm sure in real life your rollouts are much smoother than this. :P

I'm sure the NSA employees and contractors are just happy to see non-threatening post about the agency.

The downvotes probably came from people tired of the joke. It is an oft-repeated joke.

Well, I agree it got too much attention, that's for sure.

Are they all on EC2? Maybe EC2 is throwing a fit.

Amazon is, but Apple and Facebook both operate their own data centers.

Edit: I take that back, Apple uses several cloud providers, including AWS.

BGP? Tier-1 issues?

And also Azure

Apple does use S3 for storage. You didn't hear it from me.

They also use AT&T and Azure for storage too.

You did hear that from me, or anyone else with a firewall running on OS X.

...AT&T for data storage?

https://www.business.att.com/enterprise/Portfolio/cloud/

They're a massive company. When you already own telecom networks and run your own datacenters, selling cloud infrastructure services isn't too far fetched.

and Google Cloud Storage as well!

You just have to run littlesnitch, and see all the AWS servers that Apple applications hit. I don't think it's a secret that a ton of Apple applications are backended by AWS.

All hail Little Snitch. So much insight available in that app. I wish there were rule sets for Little Snitch like there are for AdBlock Plus.

lost -i in the Terminal.app will do this as well.

With a little grep-fu you could have yourself a good watchdog.

I get 'command not found' so I'm assuming I have to install something? But I can't find anything when I search Google because of the generic name.

I believe he meant to write `lsof -i`

That works a lot better! Thanks both.

oh i'm still not used to the autocorrect on OS X. Thank you for correcting my error.

Indeed, I've read multiple articles about it, some of which seem to have been fed by the Apple PR machine:

http://www.zdnet.com/article/apple-cuts-aws-spending-signs-w...

They also use METRIC TONS of ultra high-end storage.

Actually that's not saying much, because a single cabinet is around a metric ton.

So it was unintentionally accurate.

Apple discusses who it uses for cloud storage when discussing iCloud in their security guide (valid as of May 2016):

> The encrypted chunks of the file are stored, without any user-identifying information, using third-party storage services, such as Amazon S3 and Windows Azure.

[0]: https://www.apple.com/business/docs/iOS_Security_Guide.pdf

The only Apple thing that I know was on AWS was the beats1 radio thing when it launched, not sure if it still is.

Photos uses AWS for storage.

AWS status page is all green.

http://status.aws.amazon.com/

Just for the record, our company has found that Amazon occasionally omits incidents from their status page, though I doubt they could get away with omitting something this significant.

All of the cloud providers have an incentive to downplay incidents and only post them to the status page when they're really bad. The bar is pretty high to get something on there.

I mean, they outline them in their fineprint. Amazon doesn't consider the unavailability of 1 EC2 availability zone an outage because their position is that you should be architecting around that. 2 or more AZ outages in the same region and it'll show up on their status page IIRC.

"Occasionally"? <laugh>

</laugh>

Holy shit, you were laughing for 28 minutes straight. Are you alright?

You can tell who grew up in the XHTML era rather than SGML or HTML5.

Cause you know <laugh /> is just a gulp shortcut for <chuckle> </chuckle> (which is fully IE6 compatible)

XHTML 1.0 Strict all day.

I've heard that the Doppler effect https://en.wikipedia.org/wiki/Doppler_effect is responsible for AWS status page to show up green perennially.

On a serious note, why would AWS admit that it's services were not quite working well and subject itself to potential liabilities.

AWS reacts shamelessly even when your EC2 instance goes down on it's own. For a sample of how they react, see https://news.ycombinator.com/item?id=11822298

Reply on news.ycombinator.com