Facebook was down | Hacker News Reader

Facebook was down

facebook.com

325 pointssgloutnikov265 comments

Obviously anyone can "take credit" for things like this, but if this[1] is true, then jesus.

Edit: Since it's topical, I enjoy listening to Chuck Rossi in interviews or presentations. Releng 2014 - Keynote 1: Chuck Rossi, Release Engineering, Facebook Inc. | Talks at Google [2]

[1] https://twitter.com/lizardmafia/status/559963134006292481 [2] http://youtu.be/Nffzkkdq7GM?t=4m39s

Would be fascinated to see them take on big G.

E: not that I support these guys in any way. Just curious what would happen -- and whether we might all learn something about DDoS mitigation from such an event.

Google gets DDoSed all the time. You don't hear about it because, well, the DDoS SREs are very, very good at what they do.

SREs?

Site Reliability Engineers

Site Reliability Engineer. It's a Google (+Facebook)-specific title that is sort of like a sysadmin or devops, but instead of keeping the system up, they write code that keeps the system up. They also have a different negotiating position vs. engineering than in many other companies, eg. SREs have veto power over many architectural decisions in the code, and it's more "we'll build the system that can stay upright with a minimum of pagerstorms" vs. "you build the system and throw it over the wall to us and then we'll keep it upright through our self-sacrificing heroism."

Apple hires SREs who are actually sysadmins that occasionally code, complicating the title somewhat. This is not unique to them.

nostrademons's explanation of SRE is the correct one, IMO. The architecture is key. Engineering has to be built to allow that. It has helped me in the past to say SREs are concerned more with the operation of a service than a group of machines offering a service; it's almost like a service operations developer. When a company thinks in terms of services and abstracts the machine away, i.e., containers, scheduling, Mesos, Omega/<unnamed>, intelligent CI/CD, service discovery, now you're getting into SRE territory instead of SA territory. The architecture involvement distinguishes SRE from devops for me. You should be able to trust SRE to build services, not just run engineering output.

Teams that congeal out of Xooglers tend to preach SRE well, and there is the occasional company (Twitter and Foursquare come to mind) that applies the title and interacts with the team as intended.

For further reading, this is a good post: http://www.site-reliability-engineering.info/

In recent Google SREs AMA, they talked about how they mitigate DDoS attacks. http://www.reddit.com/r/IAmA/comments/177267/we_are_the_goog... http://www.reddit.com/r/IAmA/comments/177267/we_are_the_goog...

A few years ago, who would've guessed that Twitter would be the major service still standing during such a mass attack?

Before they had Mesos, they wouldn't have.

They had to leave twitter up so they could have somewhere to post?

lol. Might simply because people see white whales too often and that's not a surprise.

I am not familiar with Lizard Mafia, do they have a history of famous hacks?

They hacked Malaysian Airlines website yesterday. The wiki page goes into more detail: http://en.wikipedia.org/wiki/Lizard_Squad

That was Cyber Caliphate, not Lizard Squad.

More DDoS than hacking, but yea.

http://venturebeat.com/2014/12/30/fbi-confirms-lizard-squad-...

http://krebsonsecurity.com/2015/01/another-lizard-arrested-l...

http://www.bbc.co.uk/newsbeat/30306319

Obviously Tinder falls with Facebook, but HipChat's definitely still working for me...

Hipchat is down for me

The site is down. Not sure about the actual service.

They report issues:

http://status.hipchat.com/

What is the connection between Tinder and Facebook?

When you use Tinder, you are looking at and using Facebook profile data, as well as logging in via Facebook.

Wow.

I gotta say I'm finding it hard to accept their word that they're responsible for Facebook. I haven't paid much attention but I'm under the impression that they primarily just deal in DDoS's and other crude attacks, and I have a hard time imagining that they could cause a large enough DDoS to affect the massive juggernaut that is Facebook. Especially since Facebook just came back up and is now perfectly responsive and showing no signs of being under strain.

Yeah, it doesn't seem like a DoS since facebook was reachable and serving error pages, and instagram was reachable and serving blank pages. It _does_ seem like an intrusion or other security incident though because I'd be surprised to learn that instagram shares lots of critical infrastructure with facebook. It seems more likely that someone hit the panic button for both sites.

I wasn't even getting DNS resolution for any of the affected sites.

Some sort of security incidence does seem more plausible than a DDoS, although I can't think of what would affect Facebook, Instagram, Tinder, Hipchat, and AIM simultaneously. I'm also having a hard time imagining what sort of security incident would result in Facebook deliberately shutting down their web presence, even for a few minutes. And all of the other potential attacks, such as DNS or CDN, seems like it a) wouldn't affect everyone simultaneously, and b) wouldn't even work because sites like Facebook don't have a single point of failure, there's always backups and backups for the backups.

Tinder uses Facebook's API heavily. It's not surprising it went down with Facebook

Instagram runs within the Facebook data centers. They had some challenging scaling issues within Amazon Web Services and moved their code base to Facebook's infrastructure.

Simple Bayesian analysis says it is them as they do not have a record of lying. Why would they? There is nothing to gain and everything to lose.

Nothing to gain? Taking down Facebook seems like something that would bolster their rep significantly. If they think they can claim credit for it and get away with it, then there seems to be little reason why they wouldn't.

I guess we'll just have to wait and see if Facebook makes any public statements as to what the cause of the outage was.

    > seems like something that would bolster their rep
    > significantly

And then irretrievably kill it if they were shown to be lying.

Except that proving they didn't do it turns out to be much harder, because any amount of contrary announcement or evidence can be handwaved away as lies and misdirection. It's roughly akin to trying to talk to conspiracy theorists about 9/11.

How does simple Bayesian analysis suggest that? :-)

It might, but with horrific confidence. Someone just felt like saying Bayesian analysis.

Yeah, normal access patterns to FB are already pretty much as big as a DDOS on a regular website

For their service infrastructure, certainly. Napkin math suggest that they have to sustain peak floods in the ballpark of 500M concurrent clients. Give or take 100M.

However, if you could cripple DNS propagation/resolution from their systems to the next-hop upstreams, it would show up as services being very slow to respond. They do serve content from multiple domains.

I'd be interested to know how Google's 8.8.8.8 infrastructure defends them from DNS level manipulation...

    > they could cause a large enough DDoS to affect the
    > massive juggernaut that is Facebook

You don't need to send Facebook more traffic than it normally gets, you simply need to nail a pain point, and cause an unusual traffic event.

There's (I assume) no chance that it's been done using Slow Loris, but that's a good example of doing something unusual to deplete a resource in an unusual way.

I would tend to agree that it's unlikely Facebook have simply been flooded off the internet, but there are many other ways to perform much more targeted DDoS attacks, and presumably Facebook haven't mitigated against _all_ of them.

Good point. Although I'd say that any such attack like that is still unlikely to cause the entire site to become instantly unresponsive (or to serve static error pages as other people have suggested; personally, I just never got the site to load). And similarly, it would not be expected to come back to normal operating behavior a few minutes later. So yes, I'll grant you that you can DDoS even large sites without needing pure brute traffic, but it still doesn't seem particularly likely here.

Um, snowstorm?

I doubt it was them, though think it's worth watching since Facebook will definitely have a post-mortem. If it's not DDoS/security related, then it would be kind of embarrassing for Lizard Squad (which is why I'm surprised to see them claim it). For example, Facebook's last major downtime was caused by a main database failure – nothing to do with any vulnerabilities:

https://www.facebook.com/notes/facebook-engineering/more-det...

Edit:

I'm guessing the multiple services going down at once has to do with Akamai. It seems like there's some speculation about the storm on the east coast and their Boston datacenter.

LS have claimed responsibility for other services too though - HipChat, for example. It's unlikely all would suffer from internal issues at once.

Though, as has been said before, anyone can claim responsibility. We'll soon see.

Yeah, lets see "Blizzard Squad" try this when all of the east coast isn't trapped inside.

your Chuck Rossi was awesome! If you have other amazing videos like this to share I'm all hear!

Official update: "Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience." https://developers.facebook.com/status/issues/39399836411226...

"The issue was resolved at 23:10 PST and the site stabilized shortly afterwards. Our internal and external monitoring shows that API requests are being served with normal latency and error rates, in all geographic regions. We are sorry for any inconvenience this may have caused you and the users of your apps."

No hints there, guess we'll just have to wait for a full post-mortem to come out.

"This was not the result of a third-party attack but instead occurred after we introduced a change that affected our configuration systems."

http://www.bbc.com/news/technology-30996928

BBC[1] are reporting it was a "configuration problem", no hack

[1] http://www.bbc.co.uk/news/technology-30996928

This should be at the top.

"Never attribute to malice that which is adequately explained by a configuration problem."

Update from Facebook:

"Current State: Fix Pushed

Facebook and Instagram experienced a major outage tonight from 22:10 until 23:10 PST. Our engineers identified the cause of the outage and recovered the site quickly. You should now see decreasing error rates while our systems stabilize. We don't expect any other break in service. I'll post another update within 30 mins. Thank you for your patience."

https://developers.facebook.com/status/issues/39399836411226...

Might not be a hack! Might just be weather in Boston:

"Akamai (provider for FB, Instagram and so on) claims to be down due to power outage. #LizardSquad claims to have hacked them. Get popcorn."

https://twitter.com/petterkarlsson/status/559975340526014464

I restarted my modem and router two times before I realized that the network was fine and that it was just Facebook that was down. Props to their ops team for uptime that is so reliable.

I haven't seen Facebook go down in years. I wonder if this is going to be a big deal.

I'm wondering how much revenue gets lost every minute that Facebook is down. It has to be in the tens of thousands.

Edit: someone wrote this below in the comments, which came to $400 per second of downtime. Ouch.

$400 per second somehow seems... smaller than I would have thought. Their opex on a per second basis has got to be orders of magnitude higher than that.

$400 per second is about $13 billion per year

For comparison, the cost of downtime for a single oil rig is on the order of $100 per second. Petrobras, as an example, operates 70 rigs and has a market cap of $125 billion. Facebook has a market cap of $216 billion.

(source for cost: Cormorant Alpha shutdown in 2013)

Edit: misplaced parenthesis => my first number was way off, $70 000 vs $100. Redid analysis, made mental note not to do math before coffee intake.

That seems a tad... high.

$70,000 = $2 trillion a year, assuming 24/365 usage.

Total value of all oil produced in the world (assuming $50/barrel) is roughly $1.4 trillion a year.

You're off by an order of magnitude, 70 * 100 = $7,000, not $70,000.

http://www.offshoreenergytoday.com/cormorant-alpha-shutdown-... estimates that the Cormorant Alpha shutdown cost $10,000,000/day, or about $100/second. That figure comes from multiplying the oil production (90,000 barrels/day) by the oil price ($110/barrel). Where did you get the $70,000/second figure from?

Gah, misplaced a parenthesis. You are correct. I shouldn't do math before I've had coffee.

Facebook went down less than a year ago for a couple hours.

Restarting your modem is step 1 in troubleshooting internet connectivity problems when Facebook is down?

It wasn't stated that that was the first thing he tried. I've experienced router/modem issues where only certain sites were inaccessible until one or both pieces of equipment were restarted. I'd be interested in knowing exactly what happens to a router's software to make that happen.

I'd spitball a guess that it has to do with the firewall rules on the modem or router running themselves into a bad state. That would explain why a restart fixes it, since the state would be cleared.

That can be caused by a variety of issues. From the top of my head, I would investigate broken DNS caches and incorrect MTU settings.

It's almost like he mentioned it because he realized it was silly after the fact and wanted to share that bit of mirth with us.

Or Facebook is just that important. We will never know for sure.

It's interesting to see how Facebook downtime has a huge effect on Internet traffic – there's a Guardian report where they share their referral data while Facebook was down for 31 minutes (the longest outage in 4 years):

http://www.theguardian.com/technology/2014/jun/19/facebook-u...

Quick, Google+ Engineers: take it down too! You don't want to fall behind.

Gmail is 502'ing for me.

Gmail seems to be in working order

Google+ was actually down as well, but fortunately neither of their users tried to log in during the outage.

http://www.downforeveryoneorjustme.com/facebook.com

It's not just you! http://facebook.com looks down from here.

There's a shorter link

http://isup.me/facebook.com

It's about quality, not quantity.

http://wheresitup.com/results/54c732d88c3dcf8416232c91

All the 'is it down' websites are slowly going down with people searching for it.

"This was not the result of a third party attack but instead occurred after we introduced a change that affected our configuration systems. We moved quickly to fix the problem, and both services are back to 100% for everyone." - Facebook spokesperson [1]

[1] https://twitter.com/blakeyblogs/status/559995252485140480

Akamai, the CDN used by Facebook went down before Facebook for me this morning: http://www.downforeveryoneorjustme.com/akamaihd.net

Seems to be effecting Instagram as well. I assume they both use Akamai then.

Was using both when they became unresponsive, resetting routers all over the shop.

Looking through the customer list, most other sites seem to be working fine:

http://www.akamai.com/html/customers/customer_list.html

I wonder what makes FB and Instagram different?

It was specifically fbstatic-b-a.akamaihd.net which was also not responsive, preventing css & js to load rendering a completely broken page on my browser.

Akamai has several different networks, one for streaming media, one for regular http, and one for https. They also have an S3-like storage system and a DNS service. Some customers are configured with several of these offerings at once, so it's hard to know which, if any, experienced problems and led to the outages.

Honestly, I doubt it was an Akamai issue. If Akamai experienced network problems, dozens or hundreds of sites would be affected. If an Akamai config issue (ie human error) were to blame, then it would probably only affect one site, not several. Neither FB nor Akamai is dumb enough to push multiple site changes at once.

How much revenue are they losing every second?

$7.87 billion in 2013 / 365 days / 24 hours / 60 minutes / 60 seconds is about $250 a second.

Edit: using more up to date data:

$3,300,000,000 in the 3rd quarter of 2014 / (365/4) / 24 / 60 / 60 = $418.58

$1.5M an hour!!!

But considering this is not the peak time in North America, it's should be little less.

Yeah but I wonder if actually more people will net log in because everyone is talking about it being down.

But here in India and the rest of Asia, people wake up to Facebook. You can't ignore that.

In Asia Facebook isn't that big, they also have other social services

A site like Facebook has a different profile though. More like email. If you cannot reach your mail, you'll read it at a later point.

When facebook comes up, people will catch up on "lost facebook time" most likely. Over the course of the week (or even day) it will most likely even out.

Is the API down as well? I imagine thousands of websites and apps rely on Facebook being up.

Yes, can't use log-in with Facebook.

The main app and Messenger aren't connecting at all for me, Facebook's OAuth is also inaccessible.

Seems to be --> https://developers.facebook.com/status/

I've always thought how strange it is when people notice and make such a big deal about facebook being down or accessible. It's almost as if fb has become a public utility or something.

As others have remarked, one way one can interpret this is it is a testament to their ops' exceptional ability as engineers that such downtime is so noticeable.

>It's almost as if fb has become a public utility or something.

Well thats what Zuckerberg answered a user to the question "Why Facebook wasn't cool anymore". He didn't want it to be cool , but more like a digital utility like water or electricity

The trouble with that analogy is that water and electricity (just about) are absolutely essential to our day-to-day lives; a specific social media site isn't. Facebook needs to appeal to the trend to keep in business. Hugely popular social networking sites have failed in the past, precisely when they've fallen out if fashion.

http://youtu.be/qpNNa_Mokok?t=10s

Even the fbcdn / akamai part seems to be down (an error occured while processing your request) ... funny

Hmm, interestingly enough Faceboook today "has complied with a Turkish court order demanding the blocking of a page it said offended the Prophet Muhammad." [0]

[0] http://www.bbc.com/news/technology-30982556

This doesnt look right:

whois facebook.com @8.8.8.8

Whois Server Version 2.0

Domain names in the .com and .net domains can now be registered with many different competing registrars. Go to http://www.internic.net for detailed information.

FACEBOOK.COM.DISABLE.YOUR.TIMELINE.NOW.WITH.THE.ORIGINAL.TIMELINE-REMOVE.NET FACEBOOK.COM.GET.ONE.MILLION.DOLLARS.AT.WWW.UNIMUNDI.COM FACEBOOK.COM.LOVED.BY.WWW.SHQIPHOST.COM FACEBOOK.COM.MORE.INFO.AT.WWW.BEYONDWHOIS.COM FACEBOOK.COM.ZZZZZ.GET.LAID.AT.WWW.SWINGINGCOMMUNITY.COM FACEBOOK.COM

To single out one record, look it up with "xxx", where xxx is one of the records displayed above. If the records are the same, look them up with "=xxx" to receive a full display for each record.

>>> Last update of whois database: Tue, 27 Jan 2015 06:56:04 GMT <<<

"Facebook finally figures out that sex /still/ sells better than advertising"

It's just the way whois lookups work. Do a whois lookup of any big site (google.com, microsoft.com, etc) and you'll get a similar response.

It's been that way for a while now and happens with any large site. Try google.com for example and you'll see the same thing.

I've run WHOIS queries on FB before and gotten the same(ish) thing more than a few times -- assumed it was temporary tampering before

Appears to return hosts with name starting with search term, which doesn't make a lot of sense but has likely always been the case. Also @ flag is a dig thing, not whois.

You get the info that you're looking for if you do

whois =facebook.com

(Nobody tell him to lookup Microsoft.com)

That's why I can't comprehend companies using SaaS solutions like Slack or HipChat for something as crucial as internal communication, that requires a connection to an external server to work. What's even more scary is the data leakage risk - with DDoS you know when something bad happens, because the service simply doesn't work, but with data leakage how can you tell? Every single conversation stored on a 3rd party server. It's ridiculous. Yes, they're well protected, but it's also much more inviting for hackers - once you breach it, you gain a massive amount of sensitive data about thousands of companies... Well, it seems nice UI and 3rd app integrations are more important than security and reliability these days I guess.

If you are, say, a startup with sensitive information being discussed in an IRC server that you run, do you really think your data would be more secure then? Keep in mind that you can't afford a security expert (or your own time) to admin your system if you're a small business.

As for reliability, I've been using Slack for months without a single outage. If it went down for a day, the impact would be minimal -- I'd just use email and communicate less for that one day. Or I'd switch to another provider. Big deal.

How about P2P connections, so nothing leave your company's internal network, because why should it? It's the most secure and reliable solution, but even running an IRC server on a local machine is so easy that you don't need an admin.

Easy, yes. But very secure? No.

What about solutions like Dropbox or Gmail? Or even plain old email for that matter. Even if you look after those yourself, how can you be 100% sure that you can offer a service that is more secure and reliable than a third party?

We self-host our own HipChat servers, so they weren't affected by this.

That's cool, didn't know they offer a standalone version of the server.

You assume that every company has the knowledge and resources to provide an alternative which is more secure than what a 3rd party would be able to do as part of their core business. This, unfortunately, is not always the case regardless of the company being a startup or an enterprise...

yes. the fact that it feels so weird is testimony to their server eng team

It really is. Facebook is far from bug-free, but I'm a near-constant user and this is the first time in a long time that I can recall it going down completely. I'll be really surprised if it lasts more than an hour, at most.

Facebook did go down before, but not like this time. I tried to ping many IPs of Facebook, all even failed to reach. Can be a sign of DDoS.

Anyway, if that is true, people have to take it really seriously this time.

I mean, like, WOW. If they have done this we all must give them props for ambition.

Forget Sony. Nobody outside of hollywood bigwigs were impacted by that. But Facebook? The US government is going to go nuts. I hope lizardsquad are driving drone-proof cars.

Facebook is now back online for me on my home connection (BC Canada) and every Tor node I try. Either the attack is over or FB has fought back.

WhatsApp wasn't affected. It still runs on its own infrastructure it seems. And this might be common knowledge, but not to me.

I can get back on; able to see posts from hours ago, noting within 5 hours-ish. Anyone with an ops background able to give a JS eng who has no ops experience like myself a glimpse into what bringing services back online like that looks like? No experience with ddos mitigation techniques

Facebook status[1] on this seems dry. Hoping to see a tech blog post soon on this issue.

[1] https://developers.facebook.com/status/issues/39399836411226...

Nanog is asleep, but Ausnog might shed some light on it: http://lists.ausnog.net/pipermail/ausnog/2015-January/029463...

Looks more like a "SW update gown wrong" scenario. People reported problems a few days before already, empty timelines and such. They roll out incremently. Those kind of problems quickly affect the backends.

They are saying it was human error by facebook and hence the outage.. but how come other apps like hipchat and tinder were also down? can't be co-incidence.

FYI https://twitter.com/hashtag/facebookdown?src=tren

The only subdomain working is: http://newsroom.fb.com and hosted by wordpress.com

Yep. Down for me in Queensland, Australia as well (Facebook and Instagram). It is situations like this that give people a much needed break from social media (myself included). You don't realise just how much of a stranglehold (some more than others) that it has over you until you can't actually use it. Downtime for a site as big as Facebook is rare, but it happens, especially when your infrastructure isn't solely controlled by you (content distribution networks come to mind).

Just tweeted

"@LizardMafia: More to come soon. Side note: We're still organizing the @MAS email dump, stay tuned for that."

Oh no, I just got used to React.js/Flux

According to Twitter, Facebook falling in the dead of night does in fact make (about a few million) sounds.

There is no dead of the night on a platform with users all over the world. Maybe when it's midday over the pacific, but not even then.

Let's suspend disbelief in the name of a stretch joke.

Perhaps it's a good time to re-think our dependence on facebook for 'cheat' SSO.

... because they haven't reached five nines? Because your own privately hosted authentication system will?

Yes, it's also down in SE Asia

Bet it has something to do with this damn snowstorm. I want this season to be over with!

But http://aws.amazon.com/ and http://azure.microsoft.com/en-us/ seem to be up so some devops work needs to be done here!

Still shows Facebook is pretty popular given the widespread news its outage has caused!

I wonder how many people are calling their isp call centers right now ;)

One of our vendors crashed which hung out site. Damn counterparty risk!

yep, even ping fails. even though historically it use to work. https://bitneer/facebook.com/ping/

where's .com? ;-)

sorry about that :) https://bitneer.com/facebook.com/ping/ working now of course.

Instagram too, apparently. Are they sharing crucial infrastructure?

http://instagram-engineering.tumblr.com/post/89992572022/mig...

It doesn't help to link to the website when it is already down.

Traffic sent from HN is but a drop in a bucket the size of a dump truck for facebook.

Even instagram seems down

Makes sense. They share the same data centers. See: http://instagram-engineering.tumblr.com/post/89992572022/mig...

Funny how an outage of something as pointless as facebook generates this humongous comment count. But then, some source material for http://www.lamebook.com may have been lost...

Speculating on outages at web juggernauts is squarely within the HN remit of business and web tech.

I don't get how anyone could think that Facebook, a platform that has literally defined a paradigm shift in the way people communicate across the planet, could be genuinely considered pointless.

It's okay not to like Facebook, and there are many good reasons to be sceptical about them as a company, but to call them pointless is so utterly myopic that it beggars belief.

I was calling fb pointless from my own point of view. Obviously, lots of people disagree, otherwise it wouldn't have gotten that big. And I’m sure that every single user has their (valid) opinion on what “the point” of fb is for them. But as far as I'm concerned, I'll still call it pointless — no offence.

That'll teach you to not rely on FB to sign in to Spotify.

Back up now. Total downtime around 55 minutes that I could see.

It's down for me as well (11:50 AM IST) in Mumbai, India.

I feel strangely free.

Perhaps this is a sign that I should go back to blocking myself from Facebook with StayFocusd.

Kind of sad that I needed this kick in the backside, but I'll take this as an opportunity to reclaim empty calories and wasted time.

So of course you're on Hacker News.

Might as well take a first step to ignoring facebook. I've spent more than my fair share of time on the site in the past few years. While it's been productive overall in terms of professional and personal networking, I could have achieved as much with 10% of the time if I focused solely on the productive aspects. Two weeks ago I decided I could do without endless pointless conversations with vague colleagues, being updated on largely irrelevant minute details and thoughts of people I don't actually know, and and endless stream of low quality news links. I deactivated my account for a week and then reactivated it but only visit once a day at most. My quality of life certainly has not decreased!

Back up now. Will be good to read the postmortem on this one.

Lizard squad associated account just tweeted this https://twitter.com/LizardMafia/status/559963134006292481

True fact is that in each country it was randomly shut down

As of 11:12 PM PST Facebook and Instagram are back online.

Meanwhile world's productivity goes up substantially.

Not with everyone constantly refreshing

https://www.dropbox.com/s/zueam1oq3sdzv36/Screenshot%202015-...

Charge your battery!

"Facebook was down"

So what?

Looks like a classic Stack Snowverflow error to me.

Expensify is having issues too:

http://status.expensify.com/

I feel like this is more the norm rather than an exception for them at this point, though.

I can't wait to know the reason behind this.

Looks like cdn or part of their network routers failed, because mobile apps for instagram and facebook work fine

Back up for me, showing an error page now though.

wonder if this has to do with the storm near IAD?

Seems facebook is back. a total of 55 min or so.

It's down from here - Bay Area, California.

No one has anything else to do tonight, I guess.

Instagram as well

Looks like they share some infrastructure with facebook. Whatsap web and app works though :)

Facebook back online in SF as of 7:09 GMT

Facebook seems to be back online for me.

It's down in Eastern Europe too.

This post might as well be a poll...

with malaisian airlines getting hacked its making me think what if the ISIS got it to facebook?

Very unlikely. The attack on Malaysia Airlines was a hijacking of the DNS, not an actual hack on the website. Plus with the reosurces fb dedicates to security, they have a much better chance of finding 0days than some terrorists.

Looks like they're back online

Something to do with the blizzard?

Yep down for me too from Serbia.

Over the last fifteen years, when ever I have experienced intermittent Web access, I try www.cisco.com.

It has never been down. Ever.

The post mortem is going to be very interesting. It's Facebook, not like a Freenode server or something.

And with it millions of widgets and like buttons suddenly vanish from their webpages. Good riddance I say!

Instagram is down as well!

Here in Brazil is Down,Facebook, Tinder, Instagram, AIM and HipChat suffer decline worldwide! D:

http://check-host.net/check-http?host=facebook.com

Seems like they are back up now. 100% HA is a myth, for even those who can afford it.

It's Up and Running. All the websites are accessible now.

South African here - Down for me too.

here in Brazil, Facebook, Tinder, Instagram, AIM and HipChat suffer decline worldwide! D:

Welp...that escalated quickly.

down here (in Turkey) and surprisingly, it isn't the government this time!

Annd it's back up again

Down here (Central America)

Up again here from Norway.

It's working for me

Down from Bangladesh !

Back up for me again

Confirmed, from Iran

instagram and facebook down as of this posting

Its down in India

Yes (from India)

It's back!

Yes

down in Norway

back up

yep

Let's all have a moment of contemplation & relaxation on behalf of those poor souls on-call...

They are brave men and women for sure.

down in Singapore

yes, @korea

yup

OH MY GOD

yes @korea

yes

Down for me too.

yes

yeah

yes

Yup

yes (Central Europe)

Lizard squard???

one of the comments in thenextweb article is going nuts on the upvote/like

Affirmative

yes - Sydney, 5:59pm

Yeah

Instagram is also down

Seems Facebook isn't very webscale!

what's facebook?

Yep, it's down here in SE Asia too

DOWN for me too.

Ok they're back for me in Alberta! What a wild ride. I pooped everywhere.

Somewhere in the world Jesse Eisenberg is throwing an epic fit

ISIS hacked into facebook network ?

are you joking or do you actually have any basis for this suspicion?

Yup. Down.

http://www.downforeveryoneorjustme.com/facebook.com

I blame global warming. Them servers were too hot I betcha.

Yea, it could be Lizard Squad. https://twitter.com/LizardMafia. Facebook, Instagram, Tinder, AIM, Hipchat #offline #LizardSquad

Or it could be a power issue brought on by this little bigger than average blizzard that will hit the Boston area the hardest where, incidentally, Akamai is located.

But sure, couple of script kiddies might be responsible as well. It's usually one or the other.

Was completely dead in Australia. Down on both my local ISP and VPN to the National Research network. It was not a DNS issue as the IP Address is the same as it was during the outage.

"Host unreachable", maybe they took out the load-balancer?

- traceroute to star.c10r.facebook.com (2a03:2880:f00c:6:face:b00c:0:2)

1 redacted 2 ge-1-0-0.bb1.a.syd.aarnet.net.au (2001:388:1:5001::1) 107.649 ms 91.569 ms 91.814 ms 3 ae9.pe2.brwy.nsw.aarnet.net.au (2001:388:1:88::1) 91.981 ms 92.822 ms 103.957 ms 4 ae5.pe1.brwy.nsw.aarnet.net.au (2001:388:1:87::1) 92.447 ms 93.29 ms 101.182 ms 5 et-1-1-0.pe1.rsby.nsw.aarnet.net.au (2001:388:1:66::1) 94.606 ms 109.654 ms 91.987 ms 6 et-0-3-0.nsw-msct-bdr1.aarnet.net.au (2001:388:1:a3::2) 96.119 ms 92.032 ms 92.57 ms 7 6453.syd.equinix.com (2001:de8:6::6453:1) 99.204 ms 111.52 ms 190.499 ms 8 if-xe-0-3-1.3.thar1.1MH-Sydney.ipv6.as6453.net (2405:2000:ffd0::a) 90.392 ms 90.862 ms 92.174 ms 9 if-3-0-0.2.core1.PV4-Piti.ipv6.as6453.net (2405:2000:ffd0::1a) 232.07 ms 210.376 ms 163.293 ms 10 if-xe-3-1-1.10.tcore1.TV2-Tokyo.ipv6.as6453.net (2405:2000:ffb::22) 191.668 ms 200.129 ms 226.642 ms 11 if-ae2.2.tcore2.TV2-Tokyo.ipv6.as6453.net (2001:5a0:2200:300::2) 192.905 ms 191.953 ms 201.681 ms 12 if-ae6.2.tcore1.SVW-Singapore.ipv6.as6453.net (2405:2000:ffa0:100::49) 288.696 ms 279.671 ms 269.912 ms 13 if-ae11.2.thar1.SVQ-Singapore.ipv6.as6453.net (2405:2000:300:100::d) 264.938 ms 264.522 ms 266.912 ms 14 2405:2000:300:100::16 (2405:2000:300:100::16) 354.279 ms 349.7 ms 389.75 ms 15 ae2.bb02.sin1.tfbnw.net (2620:0:1cff:dead:beef::84c) 349.649 ms 348.205 ms 347.904 ms 16 ae0.bb01.hkg1.tfbnw.net (2620:0:1cff:dead:beef::1bdc) 349.552 ms 356.726 ms 348.74 ms 17 be6.bb01.pdx1.tfbnw.net (2620:0:1cff:dead:beef::5dd) 366.212 ms 368.99 ms 368.89 ms 18 be9.bb01.prn2.tfbnw.net (2620:0:1cff:dead:beef::f5) 365.81 ms 366.327 ms 366.598 ms 19 ae10.dr02.prn1.tfbnw.net (2620:0:1cff:dead:beef::1c27) 377.387 ms 365.613 ms 365.392 ms 23 * * * 24 ae10.dr02.prn1.tfbnw.net (2620:0:1cff:dead:beef::1c27) 387.156 ms !H 383.877 ms !H 374.672 ms !H

If they guy is posting pics of himself on Twitter asking others to "dox him", it could mean that he's pretty emotionally stable not to fall for the common pitfalls when it comes to online hacking.

In that sense, it's very possible that he has access to some major database dumps that the public is unaware of, and given that he is also claiming the glory behind the Malaysia Airlines hack; it's clear that he does more than DDoS, no matter what the more educated hackers are saying.

A long way to go for a secure internet, that's for sure.

Reply on news.ycombinator.com