> This is fucking bananas. For nearly a decade, Facebook has been at the forefront of innovating how code is deployed at global scale. They presumably have gradual rollouts, automated rollbacks, anomaly detection, not to mention (I assume) loads of organizational safeguards in place to ensure this sort of thing never happens.

> Something else happened. This was not a configuration issue. Edit: If it was, I'd expect a post-mortem post-haste.

I've worked on automation projects at a large scale, and Facebook uses an unusual and clever method to deploy their software: BitTorrent.

I can only speculate about why FB went down yesterday. But if you understand that it's being deployed via BT, you can see that there's the potential to have a lengthy rollback window.

IE, this isn't like uninstalling a single RPM; this could have impacted a significant fraction of their fleet of systems, across multiple datacenters, and if so, the amount of data they'd need to move to rollback could have been tremendous.

https://www.quora.com/Why-does-Facebook-use-BitTorrent-to-co...