Hacker News Reader

This really doesn’t work in practice. We brought down our whole compute cluster once when someone ran update and it changed some stupid thing.

It will happen to you sooner or later also. Updates are always out of band for this reason.

That’s why everybody does builds and isolates isolates updates to that process.

I feel like we’re close to that with automated testing and rollbacks. Still, it seems like a ton of complexity for what feels like a fundamental need

> We brought down our whole compute cluster once when someone ran update and it changed some stupid thing

I've seen this happen twice now, as well.