Move Fast and Fix Things

> ...that is a high-risk approach that is usually without correspondingly high payoffs

That's probably true, but it's also true that over a long enough timescale (100 years, to trigger a reductio ad absurdum) there is a very high risk that not replacing or rewriting that code will sink your technology and possibly your organization.

Just because the risk will be realized in the long run doesn't mean it's not a risk. And if the worst-case scenario is death of the entire organization, then the math could very well add up to a full rewrite. Most business managers are not prepared to think strategically about long-term technical debt. It's the duty of engineers to let them know the difference between "not now" and "never". And the difference between "urgent" and "low priority".

This is as true as it can get. I'd like to reemphasis the bug-for-bug. Sometimes that's the only way you can drop-in replace a system.

I think what works even better is to have "permission" to gradually change both the old and the new. It can drastically simplify the process of creating a replacement, if you're only replacing a slightly more sane version of the original instead of the actual original.

This is probably very context dependent, because I've learned the opposite.

For example, I was rewriting/consolidating a corner of the local search logic for Google that was spread throughout multiple servers in the stack. Some of the implementation decisions were clearly made because of the convenience of doing so in a particular server. But when consolidating the code into a single server, the data structures and partial results available were not the same, so re-producing the exact same logic and behavior would have been hard. Realizing which parts of the initial implementation were there because it was convenient, and which were there for product concerns let me implement something much simpler that still satisfied the product demands, even if the output was not bitwise identical.

You can break this down even more.

As we speak, I'm "replacing" old code by just writing a wrapper around it with the new API it should have.

Then I'll rewrite it without the wrapper, bug-for-bug.

And then I'll actually fix the bugs.

There's always more use cases for the existing code than you think there are.

There's always more corner cases handled by the existing code than you think there are.

There's always more bug fixes in the existing code than you think there are.

Combine all of those, and writing a replacement is always much harder than you expect it to be.

The biggest issue with new things is unknown unknowns.

I've learned a variation on this theme, which is more specific:

You have some code that, for whatever reason, you think is not very good. There is a reason why you have ended up with code that is not very good.

If your action is to sit down and write it all again, you should anticipate getting a very similar result (and this has been the outcome of every "big rewrite" effort I've ever seen: they successfully reproduced all the major problems of the old system).

The reasons why this happen are probably something to do with the way you're doing development work, and not the codebase that you're stuck with. Until you learn how to address those problems, you should not anticipate a better outcome. Once you have learned how to address those problems, you are likely to be able to correct the problem without doing a "big rewrite" (most commonly by fixing them one piece at a time).

Sometimes I see people attempt a "big rewrite" after replacing all of the people, thinking that they can do a better job. The outcome of this appears to me to invariably be that the second team who tried to build the system with no real experience end up following a very similar path to the first team that did the same thing (guided by the map that the first team left them, and again reproducing all the same problems).

From these observations I draw one key conclusion: the important thing that you get from taking smaller steps is that you amplify your ability to learn from things that have already been done, and avoid repeating the mistakes that were made the last time. The smaller the step, the easier it becomes to really understand how this went wrong last time and what to do instead. Yes, the old codebase is terrible, but it still contains vitally important knowledge: how to not do that again. Neither writing new code from scratch without touching the old, nor petting the old code for years without attempting to fix it, are effective ways to extract that knowledge. The only approach I've ever really seen work is some form of "take it apart, one piece at at time, understand it, and then change it".

> The hardest risk to mitigate is that users just won't like your new thing.

Do they ever? Why change the part users are used to?

Out of curiosity - when you've done this type of proxy test, what do you do about write operations? Do you proxy to a test DB, or do you have your code neatly factored to avoid writing on the test path (I guess most code I've worked on that needed a rewrite also wasn't neatly factored :) ).

That is my dream. Right now, many Windows GUIs (SmartGit/SourceTree) use 'git.exe' to manage the actual Git repositories.

If libgit2 is fully mature, I can imagine more GUIs/tools will be built to manage/analyze the git repositories.

While I think the libgit2 initiative is fantastic, I don't think there needs to be just one Git implementation.

One of my favourite things about git is that the underlying storage and protocol is really simple and straight-forward to implement. You could do a lot of it in shell scripts, if you wanted to.

The stateless storage is simple and consistent, but the thing that does vary is the various operating algorithms: diff, merge, garbage collection, etc.

This creates a really interesting ecosystem where you could potentially have third-party tools that have some secret sauce producing more efficient diffs or fewer conflicting merges but still base it entirely on the open git ecosystem and remain completely backwards-compatible with all the other tooling.

At GitLab we're very grateful for all the work that has been done on libgit2 ​by GitHub and others, and we plan to move to it completely.

I would think that the operation would either (a) have to be pure or (b) be executed in two different environments. I think going for (a) is the easier approach. If you produce an output file, make a pure operation that generates the contents, then write it as a subsequent operation. Now you can test the contents against each other, but only actually write one of them.

Basically, create an intermediary object that represents your state change and test those. Then "commit" the change from control and discard the one from the experiment.

You'd only want to use pure functions in this manner. If external state is being modified, you can use a monad, or similar, to contain it.

maybe they just do everything in production. "move fast and fix things"

That is harder than just running the experiment online.

Speculation, but if they already have the infrastructure to run the test online then it was probably easier than building one-time-use tools to test backups.

Author here. 5 years ago I would have agreed with you and logged e.g. 10 million merge requests to replay them offline. But one thing I've found over the years (which may seem obvious in retrospect) is that staging environment are not identical to production. Particularly not when it comes to finding sneaky bugs and performance regressions -- the code doesn't run on the same exact environment it will run when it is deployed (it has different input behaviors, and most importantly, it has different load and performance characteristics).

The question then becomes "why would you run these experiments offline when you can run them online?". So we simply do. I personally feel it's a game changer.

If you read about what they're doing, they basically are doing that. The tests are run independently of the production code, and production is just providing the test cases.

I've long been dreaming of adding the following tagline to my resume: "Fixing bugs by removing code since 2006"

In my team's projects, code review in PRs made a mess of the history (certain devs in particular :P). We switched to a squash merge based workflow to address it, git reflow is our particular poison: https://github.com/reenhanced/gitreflow

In the pursuit to get-things-done, we forget fundamentals and more often than not its the sound fundamentals that come in handy when your product has grown beyond your 1 or 2 member original tech team.

The first thing to do is to try to minimize the scope of mutating operations: e.g, decompose a monolithic read-write operation into independent load, transform, and store. You can then easily test as much as possible (the load and the store) side-by-side. The parts that must have side effects are still hard, but at least there are fewer of them. (One variant of this would be to build your code such that you can always intercept side-effects, and then block the new code's side effects and compare with the old code)

The next thing is to try to take advantage of idempotence (and, by extension, try to make as many of your mutating operations idempotent as possible): there's nothing completely risk free, but you can at least verify idempotence for either ordering of new and old code, and if you're factored right, you can run both paths on the same input and verify they have the same side-effects and output.

Finally, making the observation that the new code must in general be backwards-compatible with the old code, and both versions need to be able to run concurrently (because this situation will always exist during deployment): in the worst case, you can always start with a limited deployment of the new path, which limits the amount of damage done if the new code is bad.

Point the control to the real database, and the experiment to an unused database. After each request through Scientist, the two databases should be identical.

Or, make the output be the SQL command used to mutate the database. If the two outputs are different, then you've found something that needs investigation.

I haven't read the whole article yet, so I might have missed something; but how do we know that people weren't reporting these issues?

I've always had to report issues to GitHub via email as they do not have a public issue tracker (something I've always found a bit ironic).

My read of the article implies that they were running the new method on the side, and comparing the results to the old method that was still running in production. They got to 100% before they actually pulled the lever on what customers would use.

Edit: Ah, I see - talking about the Git bugs, not the differences . I’m actually not surprised that “256 (or a multiple) merge conflicts” was never noticed (or at least root-caused and fixed) by the entire git community.

Wonderful ability to use a large userbase as a giant fuzzer.

We could go look at the implementation/documentation of Scientist to confirm, but nothing says the comparison has to be happening on the response thread. You could fork a new thread to run the unused implementation without impacting response time, as long as you have the server capacity.

The article mentions running both code paths in parallel, so it becomes a matter of whether they have enough capacity for an extra cost for a (rather specific) path of their code. For this specifically I would imagine the cost is negligible in the big picture, considering regular activity far outnumbers pull/merge requests.

Using a linked-list where the actual access pattern is random and an hash table is more suitable would be the most obvious, especially since this is C code.

Similar things includes C-style string functions like strlen() and so on that require iterating over an unknown length array. Caching (or better avoiding!) this work can save a lot of computing time. Example from libgit2: https://github.com/libgit2/libgit2/commit/7132150ddf7a883c1f...

In fact C-style string is a rich source of O(n) performance issues. And git is full of strings like filenames.

It's an issue where processing something took O(n) time, when an algorithm to process it asymptotically more quickly (say O(log n)) would be more appropriate.

The old code probably works properly for the average case, but will really blow up on pathological cases. My best personal example was an optimisation pass that turned out to be exponential in the worst case. On average `n` was small, so no-one cared, until a pathological case came up where compile time ballooned to over half-an-hour. Working out that the operation could be done in a linear way only shaved a fraction of a second off the average case, but brought the half-hour case down to a few seconds.

"O(n) issue" is generic for some asymptotic optimization issue. It doesn't literally mean it was running in O(n) and there was a better O(n log n) solution. It could be O(n^2) and there was an O(n) solution or O(n) when there was an O(log n) solution or O(n!) when there was an O(n^2) solution. It just means there was a poor algorithm choice / design decision for some part of the code when a more efficient option is available.

It's notation denoting the performance of an algorithm. See https://en.wikipedia.org/wiki/Big_O_notation