Not to defend facebook, but these kinds of indirect dependencies are only obvious in hindsight.
And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.
And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.
The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time.
Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like:
External power goes out for whatever reason (don't remember)
Batteries take over
Diesel generators start up
Batteries keep draining
Diesels are spinning
All Servers shut down