A Plea for More Mikado
dmathieu.com
dmathieu.com
Another tool that's useful (if you're doing email-based development at least) is always floating minor fixes and refactoring to the beginning of your series. Then the fixes and refactors can be checked in as they're reviewed, such that the final merge is much smaller.
I can imagine it not working well in some cases, for example when a failure can only be resolved by some new thing (e.g. when a function was replaced by some other function).
There also seems to be an implicit assumption that the tests are 'sufficient', whereas I'd wager that the whole reason a revert is needed is the fact they are, in fact, insufficient. The article even says so:
> Obviously, the Mikado Method cannot work if you don’t have a good and highly reliable automated test suite.
But if you have a good test suite, then there should be no problem shipping the big change in one go!
I imagine the mikado method can still work though, because when shit does hit the fan, you can just downgrade to the previous version while keeping most of the preliminary changes? And that should then reduce the cost of the revert.
What I mean is that a big refactor might take 9 months to do atomically whereas it could have been done in 3 months as a massive single PR. However I guarantee you we'd have spent 6+ months cleaning up unexpected issues that show up after merging the giant PR. In the atomic commit approach what you get at the end of the 9 months is a solidly engineered product without those issues.
I will say that on a team, issues that you encounter are that it is very hard to keep track of these work-in-progress refactors that are getting in piece by piece, and it can cause rebase hell for other committers if there is a lot of refactoring involved. Splitting one giant refactor into 3 smaller refactors is better for the product, but represents 3x the work for other team members who have to rebase all their changes every time something is merged.
My experience with these 3 month-long PRs is that they have a tendency to not get finished. Then they get stale. Then they get abandoned. Then somebody starts another one.
In one case it was finished, but the CTO was just too afraid to deploy it because it was big and it might break something. So while we were waiting for the right deployment window it got stale, meaning more conflict fixes....
I also think most people overestimate the amount of extra work involved in breaking up a refactoring. It's usually no more than ~20% more coding, and that extra work almost always pays for itself by reducing deployment risk.
Yes, one of the lessons of process engineering a la Deming is that the important thing is to get the process under statistical control, which typically means minimizing the variance inherent in a process, rather than the mean. Doing a big refactor might minimize the mean time to completion, but it will typically have a long tail which occasionally bites you and costs you dearly. The approach that minimises the variance might take longer in mean but generally when it's done it's done. Fewer unexpected surprises.
I used a very similar method to upgrade Django from a very low version to a very high version where I would periodically:
1) Upgrade the dependencies.
2) Find something that was broken and fix it so it worked on both versions - usually with an ugly if statement.
3) Merge to master and deploy.
4) Repeat 2 days later when I had another spare half hour.
Because it was a pressure cooker environment, the testing was thinner on the ground than I would have liked but mostly because this was an enormous change this whole process took about a year. I actually left the company before I finished off the upgrade and I thought this work had been abandoned.
However, my coworker picked it up after I left and finished off the upgrade in the same way.
Before I joined there were several aborted attempts to do big bang upgrade on the same thing - stale branches that just died by optimistic people who repeatedly thought they could do it all in one go.
>But if you have a good test suite, then there should be no problem shipping the big change in one go!
No, there is. If the change is large enough this is still a horribly bad idea - especially if there are structural changes to data involved. You might be able to make bigger chunks than you would otherwise, but you should still drip-feed large refactorings into production because something can always go wrong. The fewer changes you make at once, the lower chance something will go wrong and the easier it is to isolate and fix the problem if it does.
If you are starting out with a good test suite (something I find almost never exists on a project I'm joining), you should be upgrading everything via small increments and very frequently - negating the need for using this strategy on most upgrades.
> But if you have a good test suite, then there should be no problem shipping the big change in one go!
The author is referring here to unit test suites. These cannot catch every problem that may arise due to refactoring when integrating the refactored code with the rest of the code base. His method basically means that we should look for incremental integraton tests to catch such problems early.
My strategy was more like:
1. Start making a big change.
2. See what breaks.
3. Move the big change to another branch.
4. Fix one broken thing and PR to main.
5. Rebase the big change back on top of main.
Essentially trying to break off chunks of the larger problem in small commits. I’ve had about a 90% success rate with this strategy and when it works developers really appreciate reviewing the smaller commits than one monster change.
I once heard that, “complex systems are built from simple systems,” and started to view my job more as identifying and working on the simple things to let the complex stuff fall out rather than attacking the complex stuff directly.
Edit 1: one other thing I will add is that this strategy has the huge benefit of producing many smaller tasks that more of a team can work on and commit to main rather than having one engineer do the whole refactor or having folks working out of a half functional refactor branch with nonstandard git workflows.
Edit 2: I've used this strategy in codebases with and without automated/unit testing. Where automation didn't exist, I've used dedicated QA staff. QA also appreciate testing small changes that impact part of a product rather than 'retest the entire product'. In environments where productivity is constrained by QA turnaround time I've prepared changes in a staging branch and keep having QA test the latest staging branch as soon as the previous has merged.
I’ve found that having a lot of small commits in my dev branch also helps makes rebasing easier. When there are conflicts, it’s easier to reason about what atomic set of changes they might affect than the whole wad. It also makes peeling off those changes into their own branch easier if things get to hairy.
When I work in this way I'll rebase all the nits up top where possible. I like to try to get the history to tell a sort of 'story of development'. Though this can be time consuming so I wouldn't fret about it if it becomes burdensome. When it works, this has a nice impact of dividing the PR into:
1. Here is what we were trying to do.
2. Here is all the stuff we also cleaned up which is somewhat unrelated but in the hot path of these other changes.
I would like to see something like a “workspace” where even untracked files can be shelved in a named space and returned (or picked) after some operation. I could imagine a tool like yours where a snapshot of the current working set is made and a subshell with a clean repo created. Do some work, commit or not, and when the shell is exited, everything is back to the way it was (garbage and all).
This corresponds well with what I think that software development actually is: it's almost entirely an exercise in complexity management.
10 Identify the problem factors as you currently understand them.
20 Solve simple patterns in the problem to expand your understanding.
30 GOTO 10 until complete.
We split the work into six releases over six months. Separating the two systems required a totally different architecture with a significant amount of code changes in the one system.
We didn't exactly use this method step by step (I hadn't heard of it). We thought logically through all the steps, and then reversed them. There were spike branches that we used to gather all the steps, and then we did delete them. But we only used maybe 4 or 5 spike branches total. Unit tests didn't drive anything - these changes were more system-integration level, so there weren't existing unit tests we could use.
In the end, it worked, we made a major overhaul to the one system and updated the second to now be the leader on all. We built an API on the second system, so it could be called by the first when new records needed to be created.
So I'd say, conceptually this method is what we used, but really none of our to-do list was generated from unit tests. Perhaps we could have built out a ton of tests and separated it into a release every week or every day, but we felt comfortable with the monthly releases. That helped because the QA process was pretty long (a lot of features in a regulated industry). Each release they were testing for at least a week. That being said, we built out a lot of unit tests to complement the changes, and I don't think QA found any issues in any of the six releases.
As the author described it, Mikado method is recursive. You find a to-do list on the first branch, then possibly more things on the second branch, and more things on the third. We only did one "loop", maybe two, where the author might do many.
I got this by thinking about the Kent Beck quote "first make the change easy, then make the easy change".
Much of the work of a programmer is splitting up problems in to chunks that can easily fit into our very limited bulbs. This article highlights that there are two aspects to this; one is the one widely known, which is using appropriate abstractions to reduce complexity of any isolated part of the program; but the other part is keeping the changes at a manageable scale. To me this sounds like a very valid and potentially underapprechiated part of working on code with other people involved.
When working on a big gnarly feature, I'll start by hacking it in and seeing what breaks, or what prerequisites are missing. When I find a piece that's standalone and definitely needed, I'll move that to a clean branch and get it in.
I usually do blow away the messy development branch and restart from scratch, often multiple times. I generally find the important and time-consuming part is finding out what's needed rather than the actual code itself. The final code is usually small and doesn't take long to rewrite. That matches the OP's experience pretty closely.
I'm a firm believer in Gall's law (https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law):
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
Doing things step-by-step is the only way to build something that works. And while you can do that iterative development in a branch, it's almost always better to commit those iterative steps to the trunk as soon as possible, so you never have to do that massive risky merge at the end.
My very wise and experienced CTO said to me something along the lines of: "if you don't think it's possible to break a big refactor down into small steps, then you're probably not thinking hard enough". That has stuck with me.
I wrote a blog post on atomic development a while back which references the Mikado method. https://tomsouthall.com/blog/atomic-development.
But this example seems inefficient. Keep that original failing branch in a seperate clone to refer back to. You can even pull down the fix commits back to it to observe the test count improve gradually!
And with dependencies certainly in .NET and Node and probably Ruby too there are dragons in the details. Let the compiler and/or tests do that heavy lifting rather than a tree if postit notes.
For example you assume you know upgrading gem A will need changes XYZ but I bet ya there are gotchas related to when you upgrade B and C along with A. Picking the granularity level takes a nose for it but I wouldn’t waste time planning just start chipping away at the problem to help get a lay of the land.
It feels like this method is like planning your walk to the shops and creating a survey of dogshit on your path. Just get walking and look where you walk!
I guess that does point to a benefit - it decreases the desire to hold off on merging all the incremental changes until the full refactor is done. With the WIP there will always be one loose end to tie up before it is complete and you are really sure all the changes worked together. Without it, all you have is that the individual change didn't break anything so might as well merge even if there might be rework in the future.
And if turns out that that was, indeed, necessary and the right thing to do, it’s still fresh in your mind to reimplement.
(I have not tried this method myself yet)
Initially I thought this was a plea for more Mikado chocolate sticks, a sweet snack popular in Europe :-)
Certain kinds of projects benefit from these tools. To say they are "uneconomic" is not only ignorant but condescending.
It comes off like a carpenter calling a welder "uneconomic" because the welding machine takes longer to start and cost more than a hammer and nails.
But yes, I can definitely imagine cases where trying to apply this method leads to a chicken-and-egg problem.
I don't program in Rails, but, in the languages that I do/have I don't recall a situation where it was _likely_ that a function or API that replaces something deprecated in an older version, was already available in that older version.
The only case I can think of, that happens regularly, is that something would be deprecated and marked as such, with a replacement available at the point of deprecation.
That deprecated usage should be replaced before it is removed; and if we're talking about skipping multiple major versions over a long period, the replacement likely didn't exist in the older version, so this method still wouldn't work.
If you go from version 4 (say), to version 7 (say,several years later), something could have been deprecated in 4/5 and removed entirely in 6/7, you've missed the transition period between the thing being deprecated and it being removed entirely.
There's no way you're going to use the method in OP to "fix" all of the problems you'll encounter going from 4 all the way to 7 without just going to 7 and fixing everything that's broken.
It's amazing that exist people whose personal experience alone isn't enough to think that way. But well, just to be clear, no all libraries do not agree on any way of managing updates.
the answer to that problem in this approach is to do multiple step upgrades, rather than skipping
serious frameworks/languages do not remove methods in the same version that introduces a replacement, that's the point of having a deprecation mechanism in the first place
That's what I said
> That deprecated usage (of the thing being deprecated) should be replaced (in your code, with the thing that replaced it, in the language/library) before it (the deprecated thing in the language/library) is removed (from the language/library);
The example in OP was that they were upgrading major versions after-the-fact, so they've missed the transition period (say, 2 major versions, after which deprecations are removed).
In my experience it has often not been beneficial to try and upgrade through multiple versions after the fact, when instead a big-bang update provides opportunities to improve the overall structure and quality of the code, because those things that were deprecated were done for a reason, and the thing being upgraded may have improved significantly in structure, usage, performance, etc in that time.
As you do this more and more often the dependency trees will become more clear until eventually you can generally skip the "sit down and try to do X" step, or at least come close.
An example is, you had code that worked on a local server but now you want to run it in the cloud, "natively". You sit down to start doing the conversion, but discover that you had some files you wrote to disk, and to be "cloud native" they ought to live in S3. You stop the big-bang "native cloud" conversion. You work on abstracting away what you do with the files into some sort of interface (whatever your local language calls it) which backs to something that implements that interface on a local file system. (In this particular case, there's a good chance someone has already written this abstraction layer and you just need to go pick it up, but it's not out of the question to write it yourself either; often the price of the two options are comparable.) At all points in this process you can ship the results, because even if it's ugly behind the scenes due to a half-done conversion, and you can, say, read files but not write them through the new interface, it's still shippable. And you can ship it when the only implementation of the interface is to the file system. But later on, you can implement something that backs to S3 and slip it in to the interface, and even though it's likely you'll still find a thing here and a thing there that need to be tweaked, it's far less disruptive than if you had stuck with the initial "make a big PR to convert to cloud native all in one go".
It is also more work. Although perhaps less "more work" than you might anticipate. A lot of the process of discovering what surprisingly deep dependencies on file system behaviors you had and untangling all of them is really the same in both cases. The saved effort of writing an explicit interface and jumping straight to the new thing can be easily lost in the additional effort of trying to juggle the intermediate phases (which can get quite complex) and feature flags and the bugs you introduce in the process. Even if you don't have a "Mikado" philosophy, you may well still find the process I outline in the previous paragraph is the one you adopt anyhow! In which case, why not do it independently and isolated, without a big bang?
I haven't thought "Mikado" explicitly in my refactoring plans, but really, the only "big bang" rewrite I'm willing to undertake is when my back is forced against the wall and I simply have to switch languages. Though there can still be some value in trying to straddle some services across in the meantime, even that is itself often a rather large rewrite on both sides. Otherwise, there is almost always some sort of path to doing your rewrite incrementally, one step at a time, without ever stopping main development.
If you look at the graph of "value obtained over time", it's really a no brainer to adopt this approach, too. Generally these incremental refactorings also have intermediate value they deliver. Shipping that value now means you start benefiting from it before the big bang cutover. It also means you can stop at any time and just pocket the gains you've made, with no risk of having all this work go to nought. The big bang rewrite needs to be a desperate last-ditch effort, not the first thing you reach for.
Honestly, they generally aren't even as fun as developers anticipate; they may be more fun than an incremental refactor in the first month, but the eternal grind of discovering some corner case that the old code base handled and you completely whiffed on and now you need a major rearchitecting gets to be not fun in the third month and every month subsequently. Over the course of a full project I think the incremental refactors are actually more fun.
http://lucasr.org/2011/01/29/micro-commits/
https://mrcote.info/blog/2017/12/04/more-lessons-from-mozrev...
https://dev.to/rpalo/plan-your-commits
The article focuses on refactoring and migrations, but it works also for regular development. One rule of thumb is: write a one-line commit message, and then write the code described by the commit, and just that. Another rule of thumb is that refactors should go into their own commit. So while you are coding according to your one-line commit message, if there are small refactors needed along the way, commit them separately. Fix whitespace? One commit. Extract a method so you can re-use it? One commit to extract, then use the extracted method in your next commit.
Having small atomic commits makes code that is much easier to review, and much easier to test. I think it's similar to unit testing: we change how we write code in order to make it unit-testable. This is for code review: we write code with micro-commits in order to make it code-reviewable.
This method is pretty much incompatible with squash-and-rebase. Squashing throws away all the useful information of micro-commits. Instead, just merge, and the merge commit itself is equivalent to the squashed-rebased version.
> Obviously, the Mikado Method cannot work if you don’t have a good and highly reliable automated test suite.
I disagree. Of course it's always better to have an automated test suite. But in the absence of one, the only thing you can rely on is thorough code review. And with micro-commits you have a much better chance of spotting problems if the code-diff you review contains only a small self-contained atomic change. Code-reviewing a large squashed commit is all but impossible.
- Start working on a breaking change
- Realize you need to fix something else
- Commit your WIP breaking change
- Make and commit the fix
- Rebase your changes to move the fix onto HEAD (before your WIP commits)
- Push the fix
- Continue working on the breaking change
Personally, I tend to make chains of small commits as I'm working, so keeping things in normal commits makes more sense to me. I like having everything clearly linked to the branch I'm working on. It's easier for me to forget what's in a stash and what it relates to.
To me, stash just seems like a weird non-standard version of a normal commit without any major benefits.
Basically except really giant refactors (which are rare, it’s a 20 year old codebase) you just fork off a feature branch. Master really only exists for merges and the occasional get it out right now one liner.
I say all this with the caveat that our internal scripts merge master into the feature branch on commit, so it forces you to deal with diff rot early and easily.
40 years ago it was just called programming.
That should have been at the top in a disclaimer.
I have used a similar technique on a project without any unit tests.
You can still break down a feature into it's smallest parts without unit tests.
Another slightly larger task was to refactor/fix a migration framework someone who had left the company wrote to migrate version-by-version doing all components at each version at once, instead of component-by-component doing all versions for just it at once. Without Mikado it probably would have been one giant commit, with Mikado it was 3 with everything working at each step. (If we used git probably could have been more.) And the migration bugs basically went away after that, so that was nice.
The method is something I think about when things start to feel uncertain or hairy, but I try to avoid such situations, so I haven't used the method very often in the last few years. Especially the formal version. Informally, for my solo work (it works outside of coding too) it's more like a more structured TODO list -- it becomes a TODO graph, with an obvious do-this-next item (one of the leaf nodes with no prerequisites). It can help in planning and breaking things down as well, even if it's not used for the implementation work itself. A dependency graph alone is useful for a lot of stuff.
The biggest objection is always the revert step. It's similar to TDD's red-green-refactor idea. If your tests start breaking during a refactor, you didn't correctly refactor, so rollback and try again. They should only break when you write them to fail in the red stage. I don't particularly like TDD, and don't practice it, but the idea of going back to a known-working state in the face of issues is really vital rather than trying to charge forward. Yes, in a lot of trivial cases the blocker isn't worth a node in your graph and you can get away with plowing through and fixing the red squigglies in the IDE one by one and be fine, but you should resist the temptation if you're giving the method a fair go and if past experience doesn't exactly support your hubris.
In practice reverting usually isn't a big deal. The changes you're making at any time tend to be pretty small, undoing them because you learn it's not time to make them yet isn't usually a great loss. And when you come back to it after resolving the prerequisites, doing it again doesn't take much time, because every step is supposed to be pretty simple, and you already did something like it before. (You can of course save more complicated bits off in a scratch file or local branch to refer back to later, just that it's more the exception.)
Writing the code also isn't necessarily the greatest part of overall development time. Remove one unnecessary meeting and you buy yourself time to throw away and redo quite a number of changes. And the learning of the whole system you'll get is probably more valuable too, and is reinforced when you redo things a bit differently again after resolving prerequisites. Scratch work to throw away is part of work, anyway. Like, when I'm learning a new codebase, often I make edits to it that inline a bunch of stuff because so many codebases are messes of function calls going all over the place (often for good reason revealed later) and there's only so much jumping around my brain can handle before I start losing the overall context or struggling with the debugger's limitations. I undo it all later, but the learning part is what's valuable long term.
Sometimes you need to explore options. Some wisdom from Carmack:
"Focused, hard work is the real key to success. Keep your eyes on the goal, and just keep taking the next step towards completing it. If you aren’t sure which way to do something, do it both ways and see which works better.” (https://slashdot.org/comments.pl?sid=25551&cid=2774808)
Also http://sevangelatos.com/john-carmack-on-parallel-implementat...