Hacker News Reader

What it means is that they never tested their disaster recovery system, because this would have been found right away. Or, someone would have reported it and an upper level exec would have signed off on it being okay to take 14 days to restore a small subset of users.

View on HN

Again, not that simple. The customer restore procedure was almost certainly tested(and in active use as customers blow up their own data often enough). It was _not_ tested on 800 customer stacks simultaneously, as that was considered a sitewide disaster by whoever dreamed up the failure modes to test for. Meanwhile the actual whole site disaster restore plan may or may not have been tested, but it was useless for this case since some customers were unaffected and would be damaged by the whole site plan.

Then we are in agreement. Even if it were the case they had to do 800 separate recoveries (which I vaguely remember reading they couldn't do it individually), it means they never tested a large scale recovery situation and had no idea that 800 recoveries would take 2 weeks. That's a significant issue that they should have tested.

Reply on news.ycombinator.com