Facebook, Instagram and Messenger were down | Hacker News Reader

Facebook, Instagram and Messenger were down

twitter.com

101 pointsmtsx41 comments

About 2 days ago, I noticed that Facebook added multi-level reply threads for a small percent of posters, but as of this morning it seemed totally broken for me on the latest ios build. Seems unlikely to be related, but I wonder if it's possible they broke the site trying to hotfix it.

Regardless of the root cause, I'm feeling for the oncall folks whose Thanksgivings are getting ruined right about now.

I work pretty heavily with the Facebook Graph API and things have been pretty messed up ever since Cambridge Analytica and the scramble afterwards.

I've got a ticket from August about a basic piece of functionality - deleting a post via its ID - that's completely broken, and all I get is an every-few-weeks "should be fixed", "nope, it's clearly not", "ok stand by" exchange with them.

Basic stuff like "show me the user's Facebook pages" breaks for everyone for hours at a time. When we attended F8 they talked heavily about their intensive automated testing, but I don't see that much evidence of it. I suspect a lot of things just got turned off in the mad "shut down EVERYTHING" scramble and they're still working out the impacts.

If I hadn't been looking here, I might never have known. Yes, I'm bragging.

It's a thanksgiving miracle

Man, what a wasted opportunity. If it had happened on a workday, we could have seen a massive productivity surge to boost the economy.

At least families will be forced to talk to each other instead of being on FB...

There is a major outage they are aware of effecting Facebook Login, Instagram, Messenger and the Marketing API:

https://developers.facebook.com/status/issues/22872935915151...

https://developers.facebook.com/support/bugs/555889181863481...

Edit: I think there is a bug with the status page, the outage hasn't been going for the 18 hours it is showing me here:

https://developers.facebook.com/status/dashboard/

"you must log in to continue"

Hang on... to check the status of login you need to login? What a wonderful circular dependency that is!

Also curious if FB outages affect global internet traffic? Are there any relative stats? (E.g. if google to go down, it would have an economic cost comparable to a natural disaster)

There's several anecdotes internally at Facebook that get told to new starters about some instances of when facebook.com was down. One involves masses of people calling the police to report that facebook is offline (wat, why?), another involves so many people flocking to twitter to complain that fb is down that that crowd then knocked twitter offline ("How do you know if facebook is down? Check whether twitter is online" sort of thing).

I don't feel comfortable sharing the exact figures, but during my time there the cost of a few minutes of downtime on an ads-adjacent message queue cluster directly turned into millions in lost revenue, so it's safe to say that losing the capability to _serve_ ads (as opposed to charging for serving them) would have a pretty damn high economic cost to those advertisers. As Workplace gains traction, downtime there will creep into the "cataclysmic" side of the economic costs scale.

Did you count the lost revenue as revenue per minute times the number of minutes the site was unavailable?

If so, that's actually incorrect when calculating revenue loss. There will be actually be an uptick in traffic after the outage, either through people postponing their intended visit or checking if the site is down for them after it comes back.

I know that at LinkedIn we broke part of the advertisement campaign creation flow from downtime in our service for a short time, and the impact was negligible since ads were still being served, and new ad creation would be just be retried later by advertisers.

Now if you just lost the capacity to serve ads, there might be an impact, but since a lot of advertisements are display ads (to raise brand awareness over an immediate call to action), it's not clear whether or not it's a simple minutes down times impact per minute calculation.

Facebook being down is only lost revenue for Facebook. Every other company on the planet just gained in terms of productivity and revenues.

How do you figure? Don’t a lot of businesses basically use Facebook/Instagram as their main (or only) front page?

Revenue/minute * duration actually worked here because what went down was the moral equivalent of a Kafka cluster — ads were still being served, but we couldn’t log clicks/impressions over that period of time

I would also check productivity stats. :)

You mean like "productivity goes up" and "personal happyness increases" worldwide? Probably.

the scale FB operates on is staggering! Human factor, while still present, is very low. Wonder what could have happened there. They don't usually release postmortems, right? (since the general service they provide to the masses is "free", meaning minimum obligating to reporting. Would be curious to read the postmortem, analysis of what happened.

https://status.fb.com/ads/

I wonder if WhatsApp is also affected. I know they've been trying to merge its infrastructure with that of Messenger.

Nope. Working fine in India.

Edit: Even FB is fine here.

Thailand: FB, Messenger didn't work till a wee while ago. No WhatsApp service interruption though.

WhatsApp not affected.The power of FreeBSD and Erlang. ( Not sure if they are still on that stack )

Nope, nothing to do with FreeBSD or Erlang. Can't say what it is, but there are hundreds of other differences between which pieces of FB infrastructure WhatsApp uses vs. what other products use. This is in one of those hundreds. ;)

What always amazed me was the scale of this difference not being larger. Really fine engineering. ;)

Down here in LA. Happy Thanksgiving.

Down in Orlando, Happy Thanksgiving to you too.

Both work for me here in LA!

I guess GitLab are no longer the most notable outage of the day...

Ahh. happy thanksgiving!

It must be just a coincidence that PHP 7.4 was released today, and surely FB doesnt rely on PHP version updates to latest, right?

Just speculating..

Edit: that twitter thread is quite entertaining

Facebook doesn't run vanilla PHP. It runs on Hack, a language developed from PHP that runs on HHVM, a JIT VM it developed:

https://engineering.fb.com/developer-tools/hack-a-new-progra...

Right, but up until after Hackv3.3, PHP code was supported, which support ended 9 days ago.

"HHVM v3.30 will be the last release series where HHVM aims to support PHP. The key dates are...

2019-11-19: expected end of support for v3.30"

https://hhvm.com/blog/2018/09/12/end-of-php-support-future-o...

Note: I really dont believe my above speculation is related, just thought it was an interesting coincidence, at best.

Move fast and break things during major holiday sales. I wonder if some new holiday sales feature that was supposed to go live caused outage.

Move fast and break stable infra

Perfect opportunity for everyone to wean themselves off this cesspool.

Maybe so, but please don't post unsubstantive comments to Hacker News.

Good

Could you please stop posting unsubstantive comments to Hacker News?

Is it just a crazy coincidence this happened right after Trump’s signing of the new Hong Kong bill?

probably

Given most of America, except those unlucky people like me that are at work for their 14th Thanksgiving in a row, are hanging out with family not starting at the internet either watching parades or football instead of tracking political happenings that the vast majority of Americans are clueless about because we have no realistic way as an individual to do anything about even stuff at a town/county level, I'm guessing it has nothing to do with that whatsoever.

Reply on news.ycombinator.com