We built a system called „friendly fire“ that nukes a server every 10 minutes. It has changed the mindset of all engineers and made our infrastructure missile-proof.
Funnily enough it also improved our latencies a lot (which I guess is mostly due to memory leaks et al.)