I developed and maintained CI scripts for large modular C++ application 10+ years ago. Someone added `rm -rf $(SOME_TEMP_DIR)/` to global Makefile that was run before building anything. My CI scripts did not set SOME_TEMP_DIR...

Came to work the next day, nightly build still had not finished on slave servers, had errors about non-existent home folder when tried to log in.

What made it worst was that every server mounted a NFS share that contained fingerprints and binaries of different versions of software modules built on different platforms.

Killed all slaves, restored the NFS share from week old backups on tapes, tens of developers could not create new versions of software and send previous versions/patches to customers for a while.