How to make breaking changes and not break all the things | Hacker News Reader

How to make breaking changes and not break all the things

matthew.mceachen.us

40 pointsmceachen12 comments

One important point is missing from the article: make sure to have unit/functional tests to run to make sure your changes are indeed are not breaking anything.

for the record @mrm, the above comment was not from me. :) solid article though as usual.

Absolutely. Thanks. I've added Step 0 to reference this, and added an attribution at the end of the post.

One thing that surprised me when I started at Twitter was the requirement for every line of code to be reviewed by another engineer—and if you change or add implementation, you can't ship if there isn't corresponding test changes or additions. More tests isn't a panacea, of course, though. There's definitely a spectrum of good, explanatory tests and tests-that-just-cement-the-implementation-details.

The automated tests are not panacea of course. Yet, you need to have enough tests to give you confidence that the change indeed is breaking nothing. It is one story to perform a deep refactoring on a C++ or Java code with 95% tests code coverage where either compiler or tests will pickup things for you. And it is a completely another story to do a refactoring on a PHP frontend code w/o any compiler/static analyzer and w/o any tests where your best tool is search-and-replace. Even the "depth" of safe refactoring will be very different in those two cases.

Something that works in PHP's favor is that you can write very concise and clear code.

At some level you have to trust your eyes.

I've found that integration/functional tests are far more important for these sorts of base structural changes to applications, especially when it involves fundamental changes to how data is physically stored.

This is a good starting point. One thing I'd add is the need to consider the effects of table locks during changes. It's a nasty surprise when you've tested your multi-stage process in dev, only to find during production updates that a long running select is blocking, or that your change causes request to queue until you exceed connection limits.

Very good approach. Also having some controls like feature flags per customer/user is very useful. Sometimes migration takes a while and it is useful to move users gradually through those phases with ability to rollback.

Absolutely! I didn't reference it originally, but saw that it was already out in the wild, so I added a short paragraph about Twitter's solution, "Decider."

When we were switching a contained part of our codebase from Mongo to MySQL, we ran both in parallel and stored a key in Redis to let us turn off the MySQL backend (we re-used our feature flag utility for this).

This turned out to be quite useful. If we were throwing exceptions in the new MySQL code, or even just suspected the new system of causing problems, we could switch it off. We also switched it off so that we could easily run migrations without taking down the site.

Thanks for the article!

I actually wrote it for some coworkers at Twitter—they were struggling with this, and it's a surprisingly non-trivial problem to tackle.

Reply on news.ycombinator.com