Hacker News Reader

They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...

From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second.

I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.

At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal.

Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.

I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.

Thanks. Basically they do a db change inbetween rollouts with no feature yet. Clever.

Reply on news.ycombinator.com