Hacker News Reader

Comment by bombtrack | Hacker News Reader

Obviously anyone can "take credit" for things like this, but if this[1] is true, then jesus.

Edit: Since it's topical, I enjoy listening to Chuck Rossi in interviews or presentations. Releng 2014 - Keynote 1: Chuck Rossi, Release Engineering, Facebook Inc. | Talks at Google [2]

[1] https://twitter.com/lizardmafia/status/559963134006292481 [2] http://youtu.be/Nffzkkdq7GM?t=4m39s

View on HN

I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive and showing no signs of being under strain.

Yeah, it doesn't seem like a DoS since facebook was reachable and serving error pages, and instagram was reachable and serving blank pages. It _does_ seem like an intrusion or other security incident though because I'd be surprised to learn that instagram shares lots of critical infrastructure with facebook. It seems more likely that someone hit the panic button for both sites.

I wasn't even getting DNS resolution for any of the affected sites.

Some sort of security incidence does seem more plausible than a DDoS, although I can't think of what would affect Facebook, Instagram, Tinder, Hipchat, and AIM simultaneously. I'm also having a hard time imagining what sort of security incident would result in Facebook deliberately shutting down their web presence, even for a few minutes. And all of the other potential attacks, such as DNS or CDN, seems like it a) wouldn't affect everyone simultaneously, and b) wouldn't even work because sites like Facebook don't have a single point of failure, there's always backups and backups for the backups.

Tinder uses Facebook's API heavily. It's not surprising it went down with Facebook

Instagram runs within the Facebook data centers. They had some challenging scaling issues within Amazon Web Services and moved their code base to Facebook's infrastructure.

Simple Bayesian analysis says it is them as they do not have a record of lying. Why would they? There is nothing to gain and everything to lose.

Nothing to gain? Taking down Facebook seems like something that would bolster their rep significantly. If they think they can claim credit for it and get away with it, then there seems to be little reason why they wouldn't.

I guess we'll just have to wait and see if Facebook makes any public statements as to what the cause of the outage was.

    > seems like something that would bolster their rep
    > significantly

And then irretrievably kill it if they were shown to be lying.

Except that proving they didn't do it turns out to be much harder, because any amount of contrary announcement or evidence can be handwaved away as lies and misdirection. It's roughly akin to trying to talk to conspiracy theorists about 9/11.

How does simple Bayesian analysis suggest that? :-)

It might, but with horrific confidence. Someone just felt like saying Bayesian analysis.

Yeah, normal access patterns to FB are already pretty much as big as a DDOS on a regular website

For their service infrastructure, certainly. Napkin math suggest that they have to sustain peak floods in the ballpark of 500M concurrent clients. Give or take 100M.

However, if you could cripple DNS propagation/resolution from their systems to the next-hop upstreams, it would show up as services being very slow to respond. They do serve content from multiple domains.

I'd be interested to know how Google's 8.8.8.8 infrastructure defends them from DNS level manipulation...

    > they could cause a large enough DDoS to affect the
    > massive juggernaut that is Facebook

You don't need to send Facebook more traffic than it normally gets, you simply need to nail a pain point, and cause an unusual traffic event.

There's (I assume) no chance that it's been done using Slow Loris, but that's a good example of doing something unusual to deplete a resource in an unusual way.

I would tend to agree that it's unlikely Facebook have simply been flooded off the internet, but there are many other ways to perform much more targeted DDoS attacks, and presumably Facebook haven't mitigated against _all_ of them.

Good point. Although I'd say that any such attack like that is still unlikely to cause the entire site to become instantly unresponsive (or to serve static error pages as other people have suggested; personally, I just never got the site to load). And similarly, it would not be expected to come back to normal operating behavior a few minutes later. So yes, I'll grant you that you can DDoS even large sites without needing pure brute traffic, but it still doesn't seem particularly likely here.

Um, snowstorm?

They had to leave twitter up so they could have somewhere to post?

lol. Might simply because people see white whales too often and that's not a surprise.

I doubt it was them, though think it's worth watching since Facebook will definitely have a post-mortem. If it's not DDoS/security related, then it would be kind of embarrassing for Lizard Squad (which is why I'm surprised to see them claim it). For example, Facebook's last major downtime was caused by a main database failure – nothing to do with any vulnerabilities:

https://www.facebook.com/notes/facebook-engineering/more-det...

Edit:

I'm guessing the multiple services going down at once has to do with Akamai. It seems like there's some speculation about the storm on the east coast and their Boston datacenter.

LS have claimed responsibility for other services too though - HipChat, for example. It's unlikely all would suffer from internal issues at once.

Though, as has been said before, anyone can claim responsibility. We'll soon see.

A few years ago, who would've guessed that Twitter would be the major service still standing during such a mass attack?

Before they had Mesos, they wouldn't have.

Would be fascinated to see them take on big G.

E: not that I support these guys in any way. Just curious what would happen -- and whether we might all learn something about DDoS mitigation from such an event.

Google gets DDoSed all the time. You don't hear about it because, well, the DDoS SREs are very, very good at what they do.

SREs?

Site Reliability Engineers

Site Reliability Engineer. It's a Google (+Facebook)-specific title that is sort of like a sysadmin or devops, but instead of keeping the system up, they write code that keeps the system up. They also have a different negotiating position vs. engineering than in many other companies, eg. SREs have veto power over many architectural decisions in the code, and it's more "we'll build the system that can stay upright with a minimum of pagerstorms" vs. "you build the system and throw it over the wall to us and then we'll keep it upright through our self-sacrificing heroism."

Apple hires SREs who are actually sysadmins that occasionally code, complicating the title somewhat. This is not unique to them.

nostrademons's explanation of SRE is the correct one, IMO. The architecture is key. Engineering has to be built to allow that. It has helped me in the past to say SREs are concerned more with the operation of a service than a group of machines offering a service; it's almost like a service operations developer. When a company thinks in terms of services and abstracts the machine away, i.e., containers, scheduling, Mesos, Omega/<unnamed>, intelligent CI/CD, service discovery, now you're getting into SRE territory instead of SA territory. The architecture involvement distinguishes SRE from devops for me. You should be able to trust SRE to build services, not just run engineering output.

Teams that congeal out of Xooglers tend to preach SRE well, and there is the occasional company (Twitter and Foursquare come to mind) that applies the title and interacts with the team as intended.

For further reading, this is a good post: http://www.site-reliability-engineering.info/

In recent Google SREs AMA, they talked about how they mitigate DDoS attacks. http://www.reddit.com/r/IAmA/comments/177267/we_are_the_goog... http://www.reddit.com/r/IAmA/comments/177267/we_are_the_goog...

Obviously Tinder falls with Facebook, but HipChat's definitely still working for me...

Hipchat is down for me

The site is down. Not sure about the actual service.

They report issues:

http://status.hipchat.com/

What is the connection between Tinder and Facebook?

When you use Tinder, you are looking at and using Facebook profile data, as well as logging in via Facebook.

Yeah, lets see "Blizzard Squad" try this when all of the east coast isn't trapped inside.

I am not familiar with Lizard Mafia, do they have a history of famous hacks?

They hacked Malaysian Airlines website yesterday. The wiki page goes into more detail: http://en.wikipedia.org/wiki/Lizard_Squad

That was Cyber Caliphate, not Lizard Squad.

More DDoS than hacking, but yea.

http://venturebeat.com/2014/12/30/fbi-confirms-lizard-squad-...

http://krebsonsecurity.com/2015/01/another-lizard-arrested-l...

http://www.bbc.co.uk/newsbeat/30306319

your Chuck Rossi was awesome! If you have other amazing videos like this to share I'm all hear!

Wow.

Reply on news.ycombinator.com