Facebook Down? | Hacker News Reader

Facebook Down?

facebook.com

260 pointsgamerDude197 comments

Open articleView on HN

Love the status page right now. https://developers.facebook.com/status/dashboard/

PS. Instagram app/site, whatsapp.com, messenger.com, oculus.com also down, anything else?

UPDATE: Everything back up! (Even the status page)

Do they share infrastructure?

Yeah, that's what I was wondering. Strange that there is a SPOF for Facebook/Insta/Whatsapp.

I had the impression that they are still run like mostly independent companies who only share some data on the backend.

Yeah that's what I was thinking too, so this is probably DNS related.

AFAIK, I remember reading something about Instagram moving into Facebook datacentres some time ago. I believe they were on AWS before the acquisition.

Might I ask what’s SPOF?

Single Point of Failure.

Thanks

Single Point Of Failure

Given the fact that it's also down, I'd say yes

It could be some rouge admin who actually deleted Facebook.

At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.

I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue.

For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...

I guess they have enough pending stock to deter them...

How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.

It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly).

Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.

Besides, the vast majority of code reviews are 10 minutes or less.

I think you oversimplified the code merge process at Facebook. Their internal code review system (Phabricator) would automatically add tens of reviewers to your PR because you changed an important script authored by those people. Also, there are linters that would call out the use of potentially hazardous commands. Actually deploying the thing is another process that requires jumping through a series of other review hoops.

p.s. I had a short internship there.

That’s...now how any of this works. You can’t just change integrity-bearing things without FIM systems kicking in. And you’d need collusion to get something mainlined that would bypass that.

You don't need collusion - just a code reviewer not paying proper attention.

Facebook servers do not have cron installed.

Example using widely understood tooling... But there are plenty of other one line changes with similar impact

This wouldn't "delete" Facebook or many much smaller companies. It would result in maybe a small outage and get restored immediately in most cases. It's also an infra change you'd need across many systems - this isn't possible as a single change "across entire fleet".

This is not how non-trivial services work.

one could hope

Time for them to hit the gym and call a lawyer!

Like a moulin rouge admin?

You got me there :D

Anecdotally, whatsapp worked for me (message was sent and received successfully) though Facebook gave an error. I was already logged into both services.

whatsapp.com as in the web-based client, which is down.

Finally a status page that updates the moment an issue starts

Even the status page is down

Seriously, they host the status page on their own infrastructure? That's ... not smart.

Seems like Giphy is up at least, also facebook owned.

Every company's actual status page (and news feed) is on Twitter, no matter what else they say. I don't know where Twitter hosts their real status page. Probably Facebook.

Notably, Facebook hasn't updated theirs, though.

> every company's actual status page is on Twitter

Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage.

EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.

During the recent Microsoft Azure AD outage, I attempted to use the chat widget. Selecting "Technical support" or "Billing Support" both seemed to return 500 status codes while the "Sales Support" routed somewhere else and connected me to a person, perfectly fine. Presumably the sales team don't use Azure for anything.

Circa 2008-2009 when I was at FB I wrote a dashboard widget for the ops team that scraped Twitter for mentions of phrases like "Facebook down" in the past 5m. It was in use for a while.

> I don't know where Twitter hosts their real status page

What are the odds it's some IRC bot running under someone's desk?

for posterity:

  <?xml version="1.0"?>
  <!DOCTYPE html>
  <html lang="en" id="facebook">
    <head>
      <title>Error</title>
      <meta charset="utf-8"/>
      <meta http-equiv="Cache-Control" content="no-cache"/>
      <meta name="robots" content="noindex,nofollow"/>
      <style>
        html, body { color: #333; font-family: 'Lucida Grande', 'Tahoma', 'Verdana', 'Arial', sans-serif; margin: 0; padding: 0; text-align: center;}
        #header { height: 30px; padding-bottom: 10px; padding-top: 10px; text-align: center;}
        #icon { width: 30px;}
        .core { margin: auto; padding: 1em 0; text-align: left; width: 904px;}
        h1 { font-size: 18px;}
        p { font-size: 13px;}
        .footer { border-top: 1px solid #ddd; color: #777; float: left; font-size: 11px; padding: 5px 8px 6px 0; width: 904px;}
      </style>
    </head>
    <body>
      <div id="header">
        <a href="//www.facebook.com/">
          <img id="icon" src="//static.facebook.com/images/logos/facebook_2x.png"/>
        </a>
      </div>
      <div class="core">
        <h1>Sorry, something went wrong.</h1>
        <p>We're working on getting this fixed as soon as we can.</p>
        <p>
          <a id="back" href="//www.facebook.com/">Go Back</a>
        </p>
        <div class="footer"> Facebook &#xA9; 2021 &#xB7; <a href="//www.facebook.com/help/">Help</a></div>
      </div>
      <script>
        document.getElementById("back").onclick = function() {
          if (history.length > 1) {
            history.back();
            return false;
          }
        };
      </script>
    </body>
  </html>

And the png does not load.

And here are the headers:

    # curl -i https://www.facebook.com/
    HTTP/1.1 500 Internal Server Error
    X-Frame-Options: DENY
    X-XSS-Protection: 0
    X-Content-Type-Options: nosniff
    Strict-Transport-Security: max-age=15552000; preload
    Set-Cookie: ...
    Expires: Sat, 01 Jan 2000 00:00:00 GMT
    Cache-Control: private, no-cache, no-store, must-revalidate
    Vary: Accept-Encoding
    Pragma: no-cache
    x-fb-rlafr: 0
    Content-Type: text/html; charset="utf-8"
    X-FB-Debug: ...
    Date: Thu, 08 Apr 2021 21:41:49 GMT
    Priority: u=3,i
    Transfer-Encoding: chunked
    Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
    Connection: keep-alive

    <!DOCTYPE html><html lang="en" id="facebook"><head><title>Error</title>...

>>Expires: Sat, 01 Jan 2000 00:00:00 GMT

The year 2000 doesn’t sound right.

This is common, it essentially means “don’t cache this”.

Ok, I thought this was related to SSL certs and maybe wrong dates might have caused FB to go haywire.

It loads fine for me and shows no downtime or incidents. It even says "Facebook Platform is Healthy"

Yup, everything just came back online.

Now I am unable to connect at all:

    ~# wget https://developers.facebook.com/status/dashboard/
    Connecting to developers.facebook.com (157.240.206.16:443)
    wget: can't connect to remote host (157.240.206.16): Operation timed out

Try posting

Not here. Status page still does not load at all.

same, and can't visit the site at all

I like that it doesn't know that anything went wrong.

We're obviously going to take this opportunity to dunk of facebook because it's funny, but let's not forget that there are some valuable cultural assets that facebook hosts, with groups such as "Foods with threatening auras" and "Animals in predicaments."

I long ago tamed the facebook engagement algorithm by simply unfollowing everything and everyone except these legitimate, useful types of entertainment venues.

Don't forget "The Same Photo of a Gravel Pile Every Day." Almost makes the decline of Western civilization worth it.

Is there a group that saves and reuploads the same photo every day, compressing it into oblivion?

If you start one, please share a link.

There is one in VK (not mine), been going on since 2012: https://vk.com/putineveryday

"The Same Photo of Jeff Goldblum Every Day"

I love these appreciation posts so much more than the typical Facebook-hating comments you get on Reddit in contrast. Yes, we know, FB is evil, but it doesn't mean there isn't actually some good stuff on there.

Perhaps society can rebuild if this continues.

Nature is healing.

I hope they get FB back up and running within the end of the year, otherwise I may actually start missing it.

I like how you're nostalgic, yet hate fb at the same time.

Nostalgia is rarely uncomplicated

It used to be

most relationships that come to a bitter end are like this. you want to remember the good times, but the reasons you are breaking up always come front of mind.

I walked away from fb a year and half ago, and I don't miss any of it.

Hope it stays down, it would be an overall net gain for humanity.

How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?

Probably an intern doing something dumb.

The dumb thing would be giving the intern permission to bring down the whole site.

If an intern can bring down the whole presence of your company online it's not the intern who is dumb.

bingo

some startups would like to disagree

While that's perfectly true, I'm sure a smart and determined intern could cause an astonishing amount of damage in many many orgs. Hopefully not at Facebook though.

The difference is whether or not it was intentional. If someone can bring everything down accidentally, the system is broken.

Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.

An intern shouldn't have those permissions.

DNS (not necessarily this case, but man it can bring things down fast)

Probably DNS. It's always DNS ;)

or BGP, or auth.

How does it stay up as much as it does :)

Blood, sweat, tears and caffeine.

also, some arcane shell script written by a retired engineer somewhere in the early start of the company.

"And if these people stop, the world burns. Most people don't even know what sysadmins do, but trust me, if they all took a lunch break at the same time they wouldn't make it to the deli before you ran out of bullets protecting your canned goods from roving bands of mutants."[1]

1. https://www.stilldrinking.org/programming-sucks

The global outage drinking game:

bad DNS config push - 1 shot

routing loop - 2 shots

third party advertising your routes - 3 shots

power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots

Wolves ate through fiber - 5 shots

And it was a full moon - 6 shots

Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots

And service developers left the company to found a startup - 8 shots

Expired internal SSL certificate - 9 shots

Daylight savings changeover-induced database corruption - 10 shots

Windows Update - 11 shots

> Wolves ate through fiber - 5 shots

I hope it's wolves, that would create a new category in https://cybersquirrel1.com/

You forgot:

Hunter shot your fiber relay with a hunting rifle - 6 shots

6 shots? It only needed one.

depends on the gun... some people have interesting ideas on hunting.

Apple decided to shut down Facebook.

DST can be such a hassle. It affects our reports and change requests.

An adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.

5 is often broken down to “rain fade” and “backhoe fade.”

My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”

In a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart

I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.

Coincidentally, I'm in the same predicament with Facebook, itself.

I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.

Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.

isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?

Because the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.

I was going to link a related comment but you're actually the author of it!

Here it is anyway: https://news.ycombinator.com/item?id=22316230

Please, please tell me this is a true story. It's amazing.

I can only verify being unable to get the wi-fi password. Otherwise it’s all secondhand, I wasn’t on call.

Don't forget cosmic rays!

Large scale network partitioning can happen. Data center ingress/egress can go down. If you shard to multiple data centers, it's still possible for an entire autonomous system to have an interconnect with a different AS break. That is rare as hell with the actual Internet because of all the route redundancy even between ASes, but I've seen it happen on more specialized networks. For a few days last year, the entire western half of the US was cutoff from the eastern half on Raytheon's internal TS network because someone digging a hole somewhere accidentally cut the wrong fiber line.

I'll be curious about it coming back up. It's the sort of site where there's an army of people continually trying to reload the page if something isn't working.

Likely not related, but you can imagine that they are quite busy these days with two unplanned projects:

1. Retooling the underpinnings of their ad analytics to survive iOS14. Big advertisers are about to lose a lot of their insights into attribution, partially directly related to the new limitations of iOS14, and partially due to assumptions that Facebook had made about tracking and which are no longer accurate and require a big rewrite.

2. Building an internal Shopify to allow customers to click on ads and complete transactions without ever leaving their app. This already exists but is practically unusable and nowhere close to the polish of Shopify.

3) Frantically planning Zuck's next apology tour after he leaked personal data on half a billion people.

This is an Orville reference isn't it. (or at least, it could be and would be extremely related)

No, I've just picked up the pattern that we only hear from Mark Zuckerberg when he's apologizing for a grievous "oversight" involving user data/abuse* of the Facebook platform.

* Also, underhanded use as intended.

Perhaps some security patches after the data leak? https://news.ycombinator.com/item?id=26691868

I'm getting two different error pages with the same "Sorry, something went wrong. We're working on getting this fixed as soon as we can" message.

The logo is not loading on either page.

Part of me wonders if they discovered a big/new hole that led to the data leak, although Instagram being down too suggests maybe there's something happening behind the load balancers?

I'm seeing the same thing in NYC.

Winnipeg, Manitoba, Canada here.

Australia, Sydney. It’s worldwide.

Someone moved fast.

And broke everything?

Check the A/C thermostats for hidden RPi devices

I thought they banned me. Retried in an incognito window.

Time to enjoy a brief social media pause, thanks to the deity who might have been behind this!

Required clip:

https://youtu.be/wZ6ZlRnS40s?t=12

Gold!

I was thinking of this: https://youtu.be/uRGljemfwUE

Love to see it. I hate this company.

To be a fly on the wall at Facebook HQ right now...

It's empty anyway

Weird that people almost celebrating in this thread? Facebook is a tool like any other and is only 'bad' if you allow that to happen.

That's like saying that any Turing complete language is equivalent. Some languages encourage you to do things well, others do not. It was totally possible to write code with a lot of "GOTO" statements in it that was well structured, but the actual real-world results with it were bad. It may be possible to use FB in a constructive way, but that doesn't mean we are better off having it.

Social media is a square wheel. Sure, it sort of goes and if you know no alternative, it might be better than no wheel at all. But there's no good way to use it because it's fundamentally flawed.

My hammer - a tool - doesn't want me to look at more ads. No, FB is not a tool.

Downdetector is a bit more useful than facebook.com for this: https://downdetector.com/status/facebook/

Of course the first time in months I've used any of Facebook API services, and they go down mid-implementation.

Maybe you broke Facebook

Sounds like positive correlation to me.

Do it again. You know, for science. It doesn't count if it isn't repeatable.

I wish wholeheartedly that it stays down for a long-long time, long enough to loose it's relevance.

Or long enough to have an experiment which proves the negative externalities of the platform.

And of course Instagram tells me, "No Internet Connection"

Like you couldn't connect to your server dude, don't pass the blame on!

Dupe: https://news.ycombinator.com/item?id=26743971

Perhaps they had to patch some critical vulnerability like the one that exposed 533M personal records and did the right thing by pressing the red button™

Why this outage doesn't happen to twitter

This has to be one of those rare HN instances where a popular site has gone down and people are actually happy..

I can’t think of another such instance..

popular? you must be new here...

I had put popular in italics. ;)

It’s not popular here on HN, but in the rest of the world it surely is.

Popular in the same sense that tobacco is popular.

https://www.theverge.com/2021/4/8/22374499/facebook-instagra...

In Denmark right now, everything down. Was on the phone with someone (in the US) through Messenger without issues. Hung up and tried to send them a message and nothing. Instagram non-functional as well.

Anyone else get a login request email from Instagram before it went down?

I got one, immediately logged in separately and changed my password, logged back in, then shortly after everything went down...

oh no..... YOU BROKE IT!!! D:

Oh dear. Mark Zuckerbergs website, photo gallery and his SMS messenger are down.

Time to use something that doesn't scan your photo book, address book, and ultimately; everything about YOU.

I'm guessing this has do something with the recent graph api scraping?

Facebook is kinda rude now, they probably adding api bot detection and more phone number verification.

Quickly, let your FB engineering friends know. If their status page is down, their monitoring may be down as well and they're none the wiser.

Nah, the longer it is down, the better.

> If their status page is down, their monitoring may be down as well

That is not how monitoring should work. But this is also not how status pages should work.

I bet FB Workplace went down too heh [1]

[1] https://workplace.com

I'm wondering if they're still using IRC internally.

mbasic.facebook.com was also down in New Zealand. Thought it was a company firewall issue at first, but checked https://downforeveryoneorjustme.com/ and confirmed that it's happening to others.

Same here, shows: "Sorry, something went wrong.

We're working on getting this fixed as soon as we can."

I'm seeing the same message

Anyone knows what might have caused the outage? Someone working in data centers might have a clue.

Black hawk down... hmm not this one. Blue bird down.. not again.. ohh yea facebook is down :)

I hope they publish a postmortem.

Do they usually?

They don’t but they should.

Postmortems are exciting reads and they should have a section for postmortems in their engineering blog.

It was down for sure for me. I am in Los Angeles area. Error occurred message.

I can't tell if it's down or not because I block it at the router.

Everything up and working again (in Thailand at least), that was fast

Productivity around the world likely increased while this lasted.

The website is still down for me, I'm in San Francisco.

Facebook, Messenger, Instagram and Whatsapp sites all down

Looks like facebook auth is still working?

I once accidentally knocked out 65K websites during a "fix update". Back then I was mortified. Now, I just feel like that was amateur hour...

It's back!

Down again.

Any bets on what happened? Fat Fingers?

Snapd autoupdate.

Containment breach in the AI department, they are currently bombing the site from space, the outages are collateral damage.

There's a page saying something is wrong, which could just be a very efficient summary feature for the site, that is working as intended.

Now stay down.

Quickly, let’s get shit done!

It's working for me now.

germany/berlin also outage of all fb services using http protocol

Berlin. Facebook is down.

Down in Austria as well.

Whatsapp is also down!

Instagram was as well. Seems to have been global across their properties.

Back up on Tenerife

Could this have anything to do with the recent data breach?

How long before the riots and total societal breakdown?

Yay!

Oh noes

Back up

Yea

And nothing of value was lost

down it Italy as well

Hope it's down forever!

real talk can any facebook employees remind me how to get to the IRC chats

Down in Australia as well

and its back

Yup, getting oauth errors.

Also instagram, but not whatsapp

Whatsapp (the website) now, too. Whatsapp (the messenger) is still transmitting messages.

Whoa. Wonder what on earth is going on. Can confirm here in Canada, FB and Messenger are both down.

I don't understand how can you allow this to happen? I get one platform being down, but for all of your platforms to be down is not a good sign.

I only realized now that I only used Facebook platforms to communicate with my friends, and guess what I cannot reach them by Messenger, Instagram or WhatsApp.

So I am downloading Signal... I mean come on, you are literally loosing users because of this...

Reply on news.ycombinator.com