Hacker News Reader

Required reading. Have you worked as an SRE?

> Item: Rollbacks need to be possible

This is the dirty secret to keeping life as an SRE unexciting. If you can't roll it back, re-engineer it with the dev team until you can. When there's no alternative, you find one anyway.

(When you really and truly cross-my-heart-and-hope-to-die can't re-engineer around it fully, isolate the the non-rollbackable pieces, break them into small pieces, and deploy them in isolation. That way if you're going to break something, you break as little as possible and you know exactly where the problem is.)

Try having a postmortem, even informal, for every rollback. If you were confident enough to push to prod, but didn't work, figure out why that happened and what you can do to avoid it next time.

> Item: New states (enums) need to be forward compatible

Our internal Protobuf style guides strongly encourage this. In face, some of the most backward-compatible-breaking features of protobuf v2 were changed for v3.

> Item: more than one person should be able to ship a given binary.

Easy to take this one for granted when it's true, but it 100% needs to be true. Also includes:

* ACLs need to be wide enough that multiple people can perform every canonical step.

* Release logic/scripts needs to be accessible. That includes "that one" script "that guy" runs during the push that "is kind of a hack, I'll fix it later". Check. It. In. Anyway.

* Release process needs to be understood by multiple people. Doesn't matter if they can perform the release if they don't know how to do it.

> Item: if one of our systems emits something, and another one of our systems can't consume it, that's an error, even if the consumer is running on someone's cell phone far far away.

Easy first step is to monitor 4xx response codes (or some RPC equivalent). I've rolled back releases because of an uptick in 4xxs. Even better is to get feedback from the clients. Having a client->server logging endpoint is one option.

And if a release broke a client, rollback and see the first point. Postmortem should include why it wasn't caught in smoke/integration testing.

View on HN

ALTER TABLE user ADD COLUMN country; -- oops CREATE TABLE user_email (blah blah); -- process here to move data from user.email column to user_email table. This blows up because someone has a 256-byte long email address and the user_email table has a smaller varchar() ALTER TABLE user ADD state_province;