Mea culpa: How developers fix their own simple bugs differently from other devs
neverworkintheory.org
neverworkintheory.org
Changes tend to be made higher up in the stack, ultimately the UI, because that has a lower risk of breaking something else. This gets very messy very fast.
Bugs in shared code tend to be worked around rather than fixed because other code might depend on those bugs — which quickly becomes a self-fulfilling prophecy.
Some functions/methods seem to be natural magnets for these overly local changes, and as a result, can wildly grow in size over time. I once analyzed how a 50-line function had grown to 5000 lines over a few years in a series of individually justifiable changes.
All of this is a downwards spiral that’s very hard to stop once it has started. “Refactoring sprints” and other heroic efforts sadly seem to have little impact in such an environment if they don’t also radically address the engineering practices that lead to the situation.
And I disagree about this being what goes wrong with architectures. From my experience this is more to do with entropy than anything else. It takes a continual input of effort to 'tidy up' change to keep things looking neat, and this is often ignored or under-estimated until it's too late. The other reason is when the architecture isn't fit for purpose so has to be bent to fit around the problem it's being used to solve, but that's a different argument.
No more than if you're repairing some damage to a wall, you want to work the repair into the surrounding area rather than just infilling and spot painting the exact area of damage.
Architecture is often fit for purpose when it is initially written, but a change comes along which requires a minor rearchitecture. The overly local change might be to piggy-back some data on an existing structure (which has subtly or not-so-subtly different scope), or add a flag to an existing method. If it's not just routine BAU - adding yet another CRUD handler - it's somewhat unusual for a slight adjustment in design to not be desirable, IMO.
My argument isn't for architecture czars or so-titled architects to be in charge of planning changes. It's that local changes often don't include enough rearchitecture because the person making the local change doesn't have the time to try and see the bigger picture. They just need their feature implemented, their bug fixed, their OKR satisfied.
I didn't opine as to a solution, but if I had to, I'd say (1) ensure modules have clear ownership who is / are invested in long term code health, and (2) ensure modularity is crisp and sized small enough that a person or a very small team could rewrite a module in a few months. This isn't enough - there's higher level complexity in the interaction between modules - but it's a start.
I tried to work out edge cases from the beginning; to handle invalid data; to swat the bugs before they appeared.
A bug reared it’s little insectoid head.
Now I’m wading my way through every piece of code that remotely affects the issue and making damn sure that my perfect vision is realised.
On the other hand, fixing a bug in someone else’s code doesn’t have all these personal implications. If I fix the bug without causing regressions I’m a hero! After all, it’s not personal to me.
- Someone fixing their own code fixes the codebase. Someone fixing someone else's code, generally only knows enough to make a fix at a single point, to fix an observed bug.
I think the abstract is just making statistical observations and hopefully in time will develop and test hypothesis that could create a theory. Anyway, they can use my hypothesis above and give it a whirl.
When I look at commits from some other devs that often do these localized fixes, it almost always seems like the codebase deteriotes a little bit on a global scale for example because some knowledge is duplicated from some other location 500 lines away.
I think I lean on the side of focusing on the end product (the tip of the development branch) at the cost of having not the most beautiful history ever. I'm also aware that my style of developing makes it hard to develop in large teams / a product with many release lines. For my small projects that have very little collaboration, my approach seems to work, though.
Maybe there's only two ways to have a nice history: Have a shitty end product, or be a domain expert. The latter seems unrealistic to me, because working on something you already know how to do is not very satisfying.
Bite sized ideas are really easy to digest. Maybe the real complaint is that the commits aren't sufficiently independent. I know some projects mandate that the full test suite be able to pass on every commit, which is really just a way of enforcing that every commit is "complete".
For features you may split into multiple commits if things are too big but then each commit should be self-contained and complete by splitting the feature in a number of smaller improvements.
So really, commits are what they are based on the work at hand as long as they are kept to manageable sizes. I've seen code reviews for 1000+ lines all over the place. The author may know what they are doing but the reviewers are usually lost...
We're not paying by the commit, here. As long as the commits make some logical sense, you're doing the right thing and those folks don't know how to use version control effectively.
If it's their project you sort of have to follow their rules. Otherwise I'd say just ignore them!
Optimizing your development practices for being able to revert single commits seems very wrong to me. If you produce so many bugs that this is important, I'd focus on making fewer bugs.
At my current job I can't remember us reverting a commit in the three years I've been here.
We probably work in very different environments, and your views may be the pragmatic ones in your environment.
I don't think it does. Ultimately you need to review everything so if you ensure that each commit is logically coherent it actually makes reviews easier.
Otherwise, in my experience, things can get quite messy and the history very difficult to follow.
It's also impossible to predict when or if you'll need to revert a commit. If you follow good practices and have discipline then you make your life much easier if things go wrong or you need to cherry-pick something, which is quite common with bug fixes if you have several branches (e.g. one branch per release).
So we don't need to optimize for handling them.
Compared to how much it would slow things down having to try to revert or correct a commit that both fixes a bug and introduces a feature? Especially if that change involved any sort of DB schema change that had to be reverted.
Anyone who has ever gone to production with a set of changes that involved an important bug fix and introduced a high-value feature, only to find that the whole thing had to be reverted because of a bug in the new feature, re-introducing the bug to customers, knows the importance of small, separable changes.
For a contrived example, say a bug exists because a sort algorithm sometimes reverses the order of identical values. Fixing the resulting bug locally might involve handling duplicates with some extra logic, but changing the sort might result in fixing the bug as a side effect. I might argue for both because correctness should not be an accident, but we might change one fix to an assert rather than handling a case that doesn't occur any more.
One thing I hate is when people look at the library source code to figure this kind of thing out, since any implementation detail can change with an update. Assume the most hostile implementation allowed by the docs and you’re usually fine.
In case you didn't already know: Hyrum's Law. https://www.hyrumslaw.com/ Even if the source code isn't provided and nothing is documented your users will rely on every observable nuance of your actual implementation anyway.
At Google's scale (internally, for their internal software where they could in principle fire people for doing this) this means if you change how the allocator works so that now putting ten foozles in the diddly triggers a fresh allocation instead of twenty, you blow up real production systems because although this behaviour was never documented somebody had measured, concluded ten doesn't allocate and designed their whole system around this belief and now South East Asia doesn't have working Google search with the new allocator behaviour...
In protocol design Hyrum's Law led to TLS 1.3 having to be spelled TLS 1.2 Resumption. If your client writes "Hi I want to speak TLS 1.3" the middleboxes freak out so nothing works. So instead every single TLS 1.3 connection you make is like, "Don't mind me, just a TLS 1.2 session resumption... also I know FlyCasualThisIsTLS1.3 and here is an ECDH key share for no reason" and the server goes "Yes, resuming your session, I too know FlyCasualThisIsTLS1.3 and here's another ECDH key share I'm just blurting out for no reason. Now, since we're just resuming a TLS 1.2 session everything else is encrypted" and idiot middleboxes go "Huh, TLS resumption, I never did really understand those, nothing to see here, carry on" and it works.
Algorithm B sometimes fails on the output of Algorithm A. We can fix the issue by making B deal with that case, or we can change A so it never produces that output. Sometimes changing the algorithm in A just makes the problem go away, and maybe that was a good idea anyway. This seems too abstract, so I picked a slightly more specific (sorting) thing for A.
Also the "single purpose" of a commit often includes: fixing or adding issue, documenting what / why of the fix, and mentoring other devs via example or co-review.
Cleanly split the rework and the edge cases handling can become hard enough to not warrant separate commits.
So when I come back, my code is an unfinished oil painting, and my fixes will try to complete the painting.
If I bug fix someone else's code I am not trying to complete the painting - I am at best just doing restoration.
It is a long time before Inhave made enough restorations to believe I truly own the painting.
Hmm... "Restoration", eh? https://en.wikipedia.org/wiki/Ecce_Homo_(Mart%C3%ADnez_and_G... ;-p
That did fleetingly cross my mind as I was writing it. And yeah, I suspect I may have done that a couple of times - taking over someone else's codebase is not unlike watching the pilot parachute out of a flying plane with a "it's all yours now". Sometimes you have wings and engines and radio, sometimes the port wing is on upside down but the joystick is jammed starbd so it's all ok. But you really need to know that as the pilot jumps.
All in all, I think understanding someone else's codebase is like taking over writing a novel halfway through. It's why we all like a rewrite.
(And maybe why film scripts sometimes fell like they were written by committee)
"Const, motherfucker. Do you speak it??!?"
(which, I hasten to add, would have been unthinkable if it had been a bit of peer-to-peer banter, much less senior-to-junior, but as a way of winding up your boss, it was a absolute masterwork)
While this may be true of junior engineers or people working on a project that they don't actually put into production, I feel it can't be true of most people working a devops roll on a production system.
I only worked several years in devops on a small production system with a few hundred users but I learned very quickly to respect others' work and the thought they had put into their areas and to be very wary of "fixing" anything more than the described bug.
Honestly, not exaggerating, it makes my stomach churn a bit. Regressions are very real and if you don't have decent test coverage with actual integration or functional tests, you are writing regressions and you just don't know it.
Sometimes, teaching other developers why something is a bug is the best thing for your team long term.
That's why git-fu, comments and a collective memory is important. Not only we need to know what was done, but also why, when and in which conditions it was done.
I agree with the author here, but I also see some cause for alarm. When we could see it going either way, maybe it really could. As in, the result was a coincidence. Or maybe not exactly a coincidence, but caused by some very specific details.
This investigation was on java code, but will the same hold for haskell code? Closed source code? Will it be true 20 years from now? Maybe... but I also wouldn't take the answer for granted.
And, most insidious of all, maybe we as a community will look at this and decide it's a problem to be solved, and succeed in solving it.
A little embarassed, but when i was new to my team as a jr engr i totally did this intentionally.
Each function may appear to have a contract, but it's sometimes unclear if other code is relying on the apparent (mis) behavior of the code. Obviously any sort of global state (including config files, database), magic, parallelism, async, abstraction layers make this exponentially more complex.
The author knows all the things that aren't there, it's easy. The bug-fixer just has to check every single thing or risk being bitten by something spooky (e.g. a database trigger).
The classic fear that "I don't know what else uses this function" is simple enough to solve for a developer who knows how to perform static analysis.
However, many don't. Even if they do know how to find all areas where a function is being called, it's easier for the "busy" developer to make a copy of the entire function and just change the part they need.
That's the entire premise of the techniques in Working Effectively With Legacy Code: how to ensure the code you write changes the behavior that you want to change but doesn't change behavior that shouldn't. Great book.
I wonder if there is also a social aspect to this of 'politeness' in not stepping on others' work. I wonder if there are ways around that.
EDIT: Though, interestingly, reading some of the comments with people misunderstanding the title, it looks like the article is right. If it had said the opposite, I might well have rationalized it the other way.
their bug: I'll need to rewrite most of this, what the hell were they thinking?
Yes, we establish a theory of the problem and a theory of the solution. Coherent updates relate to these both.
It is easy to debug and fix when one has that 'theory' in one's head.
Too bad I cannot see examples of what this actually entails - for us one commit which fixes several issues is a complete no-go: that should be separate commits. So now I wonder whether the paper really means single commits which address multiple things, or groups of commits?
Where I'm working now, it's the opposite to your place: It's almost a no-go to use separate commits to fix separate issues in a change that is reviewed as a whole. We have been asked to use fewer, larger commits. In practice, what I see is commits with a description (if you're lucky) that describes the main change, but there are other changes mixed in which aren't mentioned in any description.
I prefer to use separate commits to fix individual issues even when making a bugfix PR that is "larger in size and scope". That way, people can see each part and I can present the fix most clearly, and I have to figure out the best order in which each fix makes sense. I usually do this even with my own private work that nobody else sees, as there is some satisfaction and clarity in dividing up related issues. I usually construct the logical set of changes after I've figured out what they are and satisfied myself they work well together, through use of "git add -p", "git rebase -i" and per-commit testing in a "curate your Git history for others" approach.
This comes from the Linux approach, where a proposed patch series must be, in principle, a sequence of commits designed for review, that could be cherry-picked or have some of them left out.
The unit of PR should be larger than the unit of commit, and commits should each be one reasonably self-contained logical change that could in principle be cherry-picked in isolation, or individually rejected.
But lots of places have a policy which requires PRs to be "squash merged", so all the changes are one commit in the end. The individual commit history is actively destroyed. There is no point putting in the effort to lay out carefully curated commits for a PR, if most of the history will be destroyed soon after, especially if reviews don't go deep.
I've been told this should be done because it "helps with rolling back changes" but I think that's incorrect for my style of commits (because they are curated into functioning logical changes already), and it's a crude approach to rollback. However, it makes sense for those people's PRs which are chronological (commits like "try X", "try Y", "aha, fixed it"), and where part way through the PR, the project doesn't even build.
When I roll back changes that have been merged, it's rare to want to roll back an entire group of changes. By the time a problem is found with a merged PR, it's probably too late to undo it all anyway as other things will have been built on top. That's when I'm most likely to find the individual commits of the PR most useful.
Consider this line (my Java is a little rusty, but let's hope not too rusty as I start a new job next week in Java) in method get_answer of the class Query...
return fastCache.answerOrEmpty(newQuery).replace(EMPTY,defaultValue);
Perhaps the author, upon reading the bug report concludes that the problem is they've calculated a new query but in this scenario it's the similar old query they ought to check in the cache instead and they write: return fastCache.answerOrEmpty(query).replace(EMPTY,defaultValue);
However a colleague who never really understood Query.get_answer() and reads the same bug report fixes the reported site of the problem, changing from answer = Query.get_answer("Some complicated query");
to... answer = Query.get_answer("Slightly different query");
Again this is a one line change, and perhaps it cures the symptom, bug closed, but the author was correct, and "Slightly different query" just happens to avoid the case where newQuery is different enough from query for the error to cause trouble.This type of change that "fixes several issues" is fixing a correctness bug and I don't really believe you that you'd refuse to accept the first one line change because it "fixes several issues". Correctness bugs are going to do that.
But if the "fix" itself were always only a single statement then it couldn't ever be larger in some cases, and then there wouldn't be have any differences in what people submit to describe. I suppose it comes down to whether one interprets the article's "...looks at single-statement bug fixes in Java..." as "[single-statement] [bug fixes]" or "[single-statement bug] [fixes]". It feels to me like you're reading it as the former, while I think it has to be the latter.
So people could well set out to fix a bug (that the authors somehow know was) caused by a single statement, and when the code is someone else's, that's all they touch; but when it's their own, they also, "while I'm at it", fix a bunch of other stuff they notice while looking for the bug. Because they're familiar with the code, so have the confidence, and feel responsible for it, so they have the desire ("Wow, this is ugly, can't leave this in").
So all those changes together could well be several regular working-branch commits. (Maybe then rolled up into a single merge/rebase commit for PR, Idunno, it wasn't quite clear to me either.)