EDIT: Commenters are reporting that this appears to only impact users that are hitting the US data center[1], and that it may only impact users through specific ISPs
1.https://news.ycombinator.com/item?id=18434457
EDIT: Appears to be back up
I started compiling a bunch of outage reports from around the web. I have another bunch stored in Pocket that I need to add.
Facebook makes ~$105k in revenue per minute [1]. Since the outage was seemingly only US-based, and it lasted ~30 minutes, this outage probably cost them just over a million bucks.
Pretty staggering to think about.
I have always wondered whether it is accurate to think about it like this. It sure is interesting to know how much companies make in revenue per minute, but it doesn't necessarily 100% translate to lost revenue in an outage.
For example, if I wanted to purchase something on Amazon, but the retail site went down I would not visit another site to purchase it. I would just wait until the site came back up. In that sense, the revenue isn't lost just delayed.
However, Facebook doesn't have a 100% fill rate of high value ads, so during the rest of the day they'll be showing somewhat more expensive ads to users, earning a higher average revenue. That will make up for some of the loss, despite lower usage.
They lost 30 minutes of primetime eyeballs from tens or hundreds of millions of people that would have seen ads but didn't.
Iowa (direct) and SF (VPN) were both erroring out sitewide. Anecdotally it seemed to start in Analytics and spread to Facebook.com and finally to the platform over the course of ~15 minutes. But not sure if that's representative.
These kinds of outages are pretty typical for "move fast and break things" development -- instead of spending huge effort making sure that no outage ever happens, spend effort testing and limiting outages to a small part of production. The intended benefit is much faster development.
Facebook Platform is Healthy
Since October 23 at 4:10 PM