Hacker News Reader

Comment by thanatos_dem | Hacker News Reader

I’m at one of the big 5, so everything goes through code review now, but since we’re a small team, we do monitoring ourselves, and as a result we have a lot of ways to bypass the process in case we need to get a high priority hot fix out.

When I was doing a small startup, we had a small phabricator installation which I loved using. It’s very engineer-focused in a lot of its tooling and features, but has an integrated task management systems and wikis, and tools for pre-commit reviews, or post commit reviews/audits, and I found it much easier to run and maintain than github enterprise or the Atlassian suite (and it’s free).

View on HN

Why bypass the code review for high priority fixes? That's when you need it the most.

Depending on what the issue is you might want a quick fix right away which can sometimes be hacky and then put in a long-term fix that is code reviewed by others to make sure it’s the right solution.

The big example is config changes during ongoing incidents. You may need to flip a flag or update a config right now, but waiting for review, even if it takes just a few minutes, can blow slas.

There's ways to handle that somewhat though.

Also at a big 5. We've found that skipping reviews causes more problems than it solves in emergency situations. It's really tempting to bypass the process because you're sure you can fix it fast, but this mindset results in more missteps in aggregate.

If a couple minutes really makes that much of a difference you should probably page multiple people in from the start of an issue.

Generally speaking, your un-reviewed change isn't a "fix", but something to stop the bleeding. It might move traffic out of a problematic area, or disable an experiment that's leading to unusual metrics.

In other words, its a rollback, not a fix-forward. (and I'll note that where I work, these changes still need to be reviewed soon after they're submitted). I agree that all fixes should be reviewed before submission.

This is absolutely the wrong thing to do. When mitigating issues on call anything that goes live bypassing canary has to receive much more scrutiny, not less. It is best not to do such things, but if you have to do them, review the change extra hard.

See my other comment to a sibling expressing a similar sentiment. Rollbacks and drains may require submissions to change configurations, but aren't bypassing canary, they're specifically reverting to known good systems.

Reply on news.ycombinator.com