Ask HN: How to learn about safety systems and closed loop systems?

Recently, we've had a couple of failures at work that were caused by chain reactions of unmonitored things. We do our best to try and monitor everything, but really, I would like us to engineer for failure modes.

After reading the excellent writeup on partitioning of the European Power Grid, and more recently about automotive CAN networks, I'd like to learn about writing software for safety systems.

Where is a good source of the basics? Thank you!