There is a great story in Ed catmull's new book creativity inc that tells how the handled the time when rm -rf ~/* was run on the Toy Story II servers, deleting 90% of the finished film weeks before the release. Not a problem till they realized the backup system failed. I won't spoil how it ended but they recovers (not technical solution) but shows how to handle random screw ups. The book is a great read in general as it showcase their techical achievement as well as how to manage a creative culture.