Another time, one of the Engineers was doing some analysis in the MongoDB production Database, and found it easier to setup a tunnel to 27017 in his computer to connect to the DB directly. While multitasking, he decided to run the test suite of some changes he was doing to one of the programs.... but that test had as a first step to delete all tables from your local Mongo database. Next morning when I come back Ops people tell me that "the system is empty". My first thought was "What do they mean the system is empty? maybe one of the dashboards failed to update or something". My surprise when I login and see what really happened...oh boy. We spent like 5 or 6 hours getting everything back. Fortunately a combination of Backups and MongoDB replication logs allowed us to recover everything.