The Facebook Status Page is up again | Hacker News Reader

The Facebook Status Page is up again

status.fb.com

49 pointsmritzmann29 comments

Aaaand this is why you host status pages on completely distinct infra, including domain names.

There's "separate infrastructure" such as separate domain names, and then there's "host the status page on AWS".

Normal "separate infrastructure" would have failed here. And an AWS-hosted status page wouldn't have been of any use; apart from "everything is down" the status page wouldn't have known anything either. Simply having the status page be down is all the signal necessary for such a catastrophic outage.

It can be nice to know what’s going on, or at least provide ETAs if known. At least that’s what I do for my users when things go down.

Not sure if FB would provide any actionable info, but can be nice to know if you should hang around for the service to return or take a nap to be ready for when it does return.

If you're as big as Facebook, and your outage is this bad, the only type of communication to do is having a Vice President talking directly to CNN or the New York Times.

> Normal "separate infrastructure" would have failed here.

Uh, no? A status page on truly independent infra would not have shared the same BGP announcements.

> And an AWS-hosted status page wouldn't have been of any use

Well that's a matter of opinion, but I would think it's certainly better than having to provide updates via Twitter.

In this modern era we've got things like downdetector that I've found to be quite a bit more useful that status pages - if lots of people are bitching about your servers being unresponsive you're probably not being trolled most of the time.

Will they learn? No.

Is the status page always only limited to facebook for business properties? Seems like 90+% of their services are missing from this page

Facebook seems to be coming back up.

Which makes sense. Not much was down. Just DNS and BGP.

As soon as they get past the login transient, things should be OK.

Not necessarily. Depending whether internal systems have been affected, say if their internal DNS went down along with these, many other things might be affected.

Applications and appliances that have exponential backoffs might be minutes or hours away from a recovery without manual intervention; other things might be in a broken state for having given up e.g. mounting a network drive, etc.

Failures, especially on such fundamental networking technologies like DNS and routing, and especially those that take a long time to resolve, have a tendency to cascade into catastrophic failues in many other systems.

Seeing this also.

Guess now it resolves they'll be dealing with a thundering herd.

Came to say the same thing about 2020 date! The source of the document is very different to anything else FB produce. They are handling the layout with this js code for instance.

      if (window.innerHeight < 80 || window.innerWidth < 80) {
        // Blank if window is too small
        document.body.style.display = 'none';
      };
      if (window.innerWidth < 200 || window.innerHeight < 150) {
        document.getElementById('back-link').style.display = 'none';
        document.getElementById('help-link').style.display = 'none';
      };
      if (window.innerWidth < 200) {
        document.getElementById('sorry').style.fontSize = '16px';
      };
      if (window.innerWidth < 150) {
        document.getElementById('promise').style.display = 'none';
      };
      if (window.innerHeight < 150) {
        document.getElementById('sorry').style.margin = '4px 0 0 0';
        document.getElementById('sorry').style.fontSize = '14px';
        document.getElementById('promise').style.display = 'none';
      };

When a backend engineer gets asked to do some frontend work...

They've had that 2020 for all of their outages this year. I was seeing it before it came back up partially, and now I'm getting a new error page design with 2021.

Well, and all their systems presumably being disconnected from each other for the better part of a day.

The Facebook website is also partially back up. I'm seeing a few status updates, but I can't load new posts by scrolling and it just generally seems broken.

Edit: Must be intermittent, on refresh I'm getting the "Sorry, something went wrong." page again.

“ Now seeing facebook.com and other addresses resolving via DNS.”

CF CTO - https://twitter.com/jgrahamc/status/1445136845986308100?s=21

Yup, whatsapp is back for me.

It was for me too, but not for everyone at work, and it seems down again, so at best it's unstable.

Too bad it’s full of lies:

“Some users”

“May be impacting”

“May be experiencing issues”

None of those are lies, they are simply unhelpfully unspecific.

I understand your frustration at imprecise communication - but if you are advocating for precise and accurate phrasing, the least you can do is follow your own advice.

To be fair, if you can 100% verifiably say that none of your users can access your website because of a very specific technical reason, "some" at best is a partially truthful description and certainly not an honest one given the information available.

I guess it really depends on how much you respect your userbase and how flexible you are with the meaning of words. ...

And we have seen time-and-again how much Facebook respects its userbase.

The classics. Another one of my favourites is "increased error rates" when a service is completely down.

"No one will notice the understatement if the problem is solved fast enough."

The usual boilerplate phrases coming straight from the masters of PR

just ran a dig A facebook.com, it's resolving and it wasn't a little while ago, obviously something is working again. same with insta and whatsapp

I love how instagram's error message is literally text in a body element "Oops, an error occured"

Facebook app, Instagram and Messenger are working for me.

Reply on news.ycombinator.com