At PyCon in Cleveland this year Amjith Ramanujam did a presentation on "how Netflix does failovers in 7 minutes flat" [0]. Worth a watch/listen for anyone interested in what their response may have looked like. Now I'm curious to read a post-mortem from them and see whether their procedures worked as expected (it sounds like they were down longer than 7 minutes) or where they encountered unexpected issues.