How to approach and prioritize technical debt
leadership.garden
leadership.garden
The hardest thing, especially with non-technical founders, is getting them to understand that yes, their precious features need to be pushed back a bit, but that this will prevent major issues, downtime, and slowdowns next quarter or next year, and that issues they are experiencing now are often due to neglecting tech debt in the past. Once they get this, the company usually has a much better outlook.
This is also an investing criteria (what is your ongoing plan to reduce technical debt) for some of the better tech investors, I've heard.
"you can have X in three months or Y in two weeks but if you pick either one, that delays Z by the same amount of time" focuses the attention.
1) It is impossible to avoid gathering technical debt. The code will deteriorate in one way or another. You need to prepare to fix it since you can't avoid it.
2) It is so extremely difficult to make a correct "risk assessment" on technical debt so you should avoid doing so at all. You will just end up arguing all day on the merits of clean code/architecture vs feature growth. Instead, keep two backlogs and reserve a set amount of resources on each, e.g. 20% on reducing technical debt and 80% for product features and other development.
3) The "cost" of reducing technical debt is actually negative. That's the whole point of working with it, to increase development velocity.
Code gets plastered over with features and abstraction layers, but that's an active process that we are complicit in doing.
More germane for this discussion, technical debt was originally defined as a positive thing that you want to go get... It is the mismatch between our domain model and how our users think about the domain. It is positive because “enough with all of the planning and interviewing and requirements gathering and careful architecting, can I just build something and have my users criticize it and do four or five drafts until I get it right?” ... The debt is the drafts before you get it right, the interest is the constant translation between the language of your users and the language that the system is expressed in. You have a “contracts” table that contains things that are not contracts because every purchase foreign keys to a contract, but your users have since wanted to know what they do with the purchases that are not associated with any contract, where do they go. And now every query that aggregates over contracts needs to exclude the non-contract contracts, this is part of the interest you are paying.
But at some point it started to mean that we had to upgrade our dependencies, that's tech debt, or this kludge that I threw in, that's tech debt, or the fact that we never worked out a shared library between the front end and the back end and so any Python code in utils.py needs to be manually kept up to date with a file utils.js in a separate Git repo so that we are sure that we can do these things both on the front end and the back end. More broadly, anything that we no longer care for is tech debt. And that's where you really get this idea that it is deteriorating, that's more a measure of our own patience deteriorating, especially as we never seem to have time for the refactors we want.
Perversely the cause of not being able to refactor has nothing to do with tech debt and cannot be solved in this way. It's multifaceted so at different places it emerges in different ways, but usually it's an incentive problem. At some places it is that the only incentives are for feature development. At other places it is that every developer is working on a different thing rather than prioritizing one thing that the business needs and delivering it, and allowing developers to use their slack time in this equation to improve the product however they see fit... In yet another it is because the team lead resists any suggestion that the framework being used is too heavyweight for the problem being solved and so trying to keep these very clean abstraction layers is causing people to have to rewrite the same basic thing in five different unrelated places, because it has to bubble up from the data layer into the service layer into the business layer into the controller layer into the API into the consumer layer into the app state layer into the frontend model layer...
It kind of does -- if you leave a codebase alone for a long time, and you come back to it later to upgrade a lot of dependencies (sometimes making a multi-version jump), it's a lot harder than it would have been to keep them updated as new versions were released.
It would have been a lot worse if that log4j CVE had been in a library with a lot more transitive dependencies or makes breaking changes between versions, like Jersey.
One advantage of monorepos with shared dependencies is that even the parts of the code that don't need to be touched very often will still get the latest dependency updates. If those codebases are in standalone repos, they just sit there, and then one day a simple attempt to upgrade a dependency turns into hours of work.
So, it's not technically "rotting," but it's definitely the case that leaving code to sit creates more technical debt later on -- even if that code was perfectly good the last time anyone worked on it.
http://c2.com/doc/oopsla92.html
Technical debt is only "positive" in the sense that it may permit shipping something now or earlier, but it accumulates and becomes something that can slow the project team down to a crawl, or worse totally stall forward progress. In the end, unless your project can be thrown away, it's a negative.
It's not just that it permits something now or earlier, but in shipping something now or earlier it can also increase the quality of what you are shipping. Ward saw it as the center of his preferred style of coordinated software development, XP. Sort of the old Daoist idea that the wheel needs the hole at its center, the negative enables the positive.
> I became interested in the way metaphors influence how we think, after reading George Lakoff and Mark Johnson's Metaphors We Live By. An important idea is that we reason by analogy with the metaphors that have entered our language.
> I coined the debt metaphor to explain the refactoring that we were doing on the WyCash product. This was an early product done in DigiTalk Smalltalk, and it was important to me that we accumulate the learnings we did about the application over time by modifying the program to to look as if we had known what we're doing all along, and to look as if it had been easy to do in Smalltalk.
> The explanation I gave to my boss, and this was financial software, was a financial analogy I called the debt metaphor. And that said that, if we fail to make our program align with what we then understood to be the proper way to think, uh, about our financial objects—then we were going to continually stumble over that disagreement: and that would slow us down, which is like paying interest on a loan! With borrowed money, you can do something sooner than you might otherwise, but then until you pay back that money, you'll be paying interest.
> I, uh, I thought borrowing money was a good idea, I thought that rushing software out the door to get some experience with it was a good idea. But that of course you would eventually go back, and as you learn things about that software, you would repay that loan by refactoring the program to reflect your experience, as you acquired it.
> I think that there were plenty of cases where people would “rush software out the door,” and then learn things, but never put that learning back into the program. And that, by analogy, was borrowing money thinking that you never had to pay it back. Of course, if you do that, say with your credit card, eventually all your income goes to interest, and your purchasing power goes to zero. By the same token, if you develop a program for a long period of time by only adding features and never reorganizing it to reflect your understanding of those features, then eventually that program simply does not contain any understanding and all efforts to work on it to take longer and longer. In other words the interest is total—you'll make zero progress!
> A lot of bloggers at least have explained the debt metaphor and uh, confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later... and thinking that that was the primary source of debt. I'm never in favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of a problem, even if that understanding is partial.
> You know, if you want to be able to go into debt that way, by developing software that you don't completely understand, you're wise to make that software reflect your understanding as best you can: so that when it does come time to refactor it's clear what you were thinking when you wrote it, making it easier to refactor it into what your current thinking is now. In other words, the whole debt metaphor, or let's say the ability to pay back debt and make the debt metaphor work for your advantage, depends upon you writing code that is clean enough to be able to refactor as you come to understand your problem. I think that's a good methodology, it's at the heart of extreme programming... The debt metaphor is one of many explanations of why extreme programming works.
—Ward Cunningham, https://youtu.be/pqeJFYwnkjE
Your specific codebase may not, but the frameworks, OS, and hardware it's running on evolves and changes overtime. Software practices change over time as we find more effective ways (hopefully) to do things. If you expect to let code sit and come back to it in 10 years and think that everything will be hunky-dory, I've got a bridge to sell you.
> It is the mismatch between our domain model and how our users think about the domain.
...but the business domain continues to evolve. A new client comes in, another software system is onboarded, the organization is restructured, new regulations are inacted that need compliance. I think the idea that technical debt only exists because we didn't "perfectly match" the domain and our model at the start is a bit short sighted.
> But at some point it started to mean that we had to upgrade our dependencies, that's tech debt, or this kludge that I threw in, that's tech debt
The broadest definition of technical debt that I've seen boils down to technical challenges that impede developer progress that does not have direct concern to the business. One of the reasons that I think that it frames a purely technical issue (i.e how the debt got there and how it needs to be resolved) with something that the business wants (rapid development). While there are a multitude of challenges in resolving technical debt, a significant one is getting the business to allocate time to resolve it. In order to do so, there needs to be some value for the business, not just the developers.
>More broadly, anything that we no longer care for is tech debt. And that's where you really get this idea that it is deteriorating, that's more a measure of our own patience deteriorating, especially as we never seem to have time for the refactors we want.
If you're defining technical debt as "anything you don't like", then you don't have a real understanding of your technical debt. One of the key pieces in the broad definition above is that the debt "impedes developer progress". Old code bases have plenty of things that modern developers don't like, but it works and it doesn't impact the application overall. It's old, but we barely touch it, so it's age or style is of little consequence. If you don't know what technical issues are causing signficant issues for your team or your codebase, then there's little justification for "paying it off". If you can sell the business on it, then you tend to get into long refactoring projects that provide little value than inflating developer egos.
Lehman's laws of software evolution 1 and 2:
"A system must be continually adapted or it becomes progressively less satisfactory. As a system evolves, its complexity increases unless work is done to maintain or reduce it."
https://en.wikipedia.org/wiki/Lehman%27s_laws_of_software_ev...
https://users.ece.utexas.edu/~perry/education/SE-Intro/lehma...
This is such a disappointing aspect of the 'technical debt' concept and surrounding discussion.
The unspoken truth is that technical debt is most often a euphemism for inexperience and lack of skill. This may be an uncomfortable truth but it doesn't have to be inhumane. It's not inhumane to identify a lack of skill and then address it.
The problem with 'technical debt' you'll won't find a concrete definition, and you won't see a consistent usage across the board. It is far more used as a way for engineers to posture as if "we meant to build it that way", we were making an explicit cost-benefit tradeoff... spend more time or get it out quickly.
Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued.
The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set.
But this incorrect. How much upstream scope/features delivered is certainly a function of time. Feature/Capability X, Y, Z will take more time than X and Y alone. But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set. Many will balk at this, I realize. Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'.
'Technical debt' hurts everyone because it doesn't admit to any of this, and shifts the buck away from any personal skill issue. So, too frequently the engineer goes back to fix the technical debt and not their own skill set. That's why there's always a perpetuating game of technical debt over years until the great 7 year rewrite.
Technical debt is the answer to management requests that cannot be accomplished in the requested time with the available resources. No more, no less. The word debt refers to the fact that the shortcut needed to do so will cause problems repeatedly in the future (interest that will need to be paid).
In my experience, engineers will make this clear to management, and will point out the additional time needed to implement a solution that avoids those problems, and management will make the decision to continue as is.
For a concrete example, management requests that you implement real-time collaborative editing in an existing tool. You have enough time to replace all the backend stuff with a CRDT or OT based implementation, but the frontend architecture is not equipped to deal with collaborative editing and there is no time to do both with the people available.
So, in order to meet the management deadline, quick hacks are added to bolt on the collaborative UI on top of the existing UI. Eventually new features have to be added that simply cannot be supported in the existing UI architecture without massive workarounds. Carving out some time to slowly refactor the UI architecture is rarely done, so the problem accumulates until development speed slows to a crawl.
That is technical debt, and it's entirely a byproduct of management/development negotiations. Has nothing to do with engineers. Nobody could have predicted that the architecture would need to support an entirely different backend in the future.
EDIT: Just to be clear, technical debt happens as a result of a conscious tradeoff between one or more options, where the more problematic option is chosen to get to an "acceptable" solution in a shorter period or with less resources. Some dev writing a shitty architecture is not technical debt. Depending on a library that later gets EOLed is not technical debt.
Figuring out you have a shitty architecture or an EOL library and choosing to continue building on it despite clear evidence it is a bad idea, in service of shorter time to market, that is technical debt.
1. By writing the needed code the "debt incurring" way, you learn how to do it the "right" way. So you may start with inexperience, but you gain the experience and are now better able to fix the issues you introduced.
2. Similar to this, even if you have all of the necessary experience and skill, you still may not be fully able to picture how the feature will need to evolve in the future to support additional use cases. Once you see that happening, you can design a better implementation, but not before then. Sometimes you have to do it wrong to get the feedback you need to do it right.
3. There are often real tradeoffs that get made that aren't about skill, but about time. I don't think you can discount this situation, in my experience it happens very, very often.
A great way to avoid nearly all of this is to just set "EOL criteria" for your software. "This project will not work beyond 100 users." or, "This feature will need a rewrite once we have to account for the edge case we ignored in this implementation." Know the limits of what you've built, and make it clear that you need to rebuild once you've reached those limits, and it's at least marginally more likely you'll get the support you need from the people with the purse strings.
> Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued.
I don't see how the second statement follows from the first. Are you saying that good engineers always have enough time to implement every feature "the right way" on the first try?
> But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set.
Of course it is a function of immediate time available today.
> Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'.
This sounds like it's targeted at one specific developer you used to work with (I have worked with developers like this too, it sucks)... but that has nothing to do with time/quality tradeoffs in design and development.
> The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set.
This is like Calvinism for programmers. Yes, some programmers are better at software architecture and design than others. Yes, some programmers leave behind a mess which, if not fixed, does truly turn into "legacy code" and "tech debt".
But it's ridiculous to assert that all technical debt follows this pattern. Design is often iterative; people are not omniscient. Bad engineers fail to evaluate the tradeoffs correctly, or aren't aware of all their options, and make bad decisions. Good engineers understand the tradeoffs and know their options, and make good decisions.
Let me give an example from my actual job:
I was tasked with implementing a sort of graph lookup system. I had two options: pre-compute a large portion of the graph, leading to a nice big flat table that could be easily queried at runtime with a single join, or leave the entire thing in a normalized format and do a big multi-join lookup at runtime. I chose the second option, because this allowed me develop a system and codebase that was simple and easy to understand, offloading the complicated work to the database. Moreover, it allowed me to avoid the various operational issues and technical details around setting up and maintaining the intermediate pre-computed table. The performance requirements were such that the increase in latency and server load wasn't important for the first version of the product. The business was and remains happy with the product.
Now that it's humming along and users are starting to put more load on it, migrating to the first option is now on our engineering calendar for the year. With ~12 months of accumulated knowledge about this system and its use cases, with an increasingly stable API, and with an increasingly rigorous & thorough test suite, I feel confident that swapping out the internals (including the database layout) is a relatively low-risk operation at this point. The internals have already been subject to several incremental refactorings simplifications as the business stabilized the product requirements; these were implemented without any visible effect on users, and with minimal changes to other "internal consumers" who depended on the API.
If I had tried to implement things that way from day 1, I would have had a system that was more complicated, more difficult to maintain, more difficult to explain to other programmers, and had more operational elements that could break. But I did indeed take a loan of "tech debt" in the form of worse performance and high server load, which we will have to pay off eventually.
Perhaps if I was a better programmer, I would have had the skills to implement the first option right, on the first try. But I don't know many (any?) programmers who do have those kinds of skills, while also having the wherewithal to document the thing, write useful and nontrivial test assertions, develop tooling for OpenAPI conformance, et alia. Not to mention that the flexibility granted by simplicity actually has turned out to be very important for being able to respond to changes in product requests quickly.
It seems (so far) that I did make the right decisions in this situation. I think that makes me at least a decent software engineer.
Instead what happens is they say "Well, build what you can for $1m", and you say "Ok, we can make 'a bridge' for that", and then either (a) your infantry can cross, but the tanks have to get diverted 20 miles out of the way, or (b) the tanks end up in the river along with the bridge. Since (b) is bad, you then have to spend a lot of time planning the routes for the tanks, and making sure the tanks have the right air cover, etc etc, i.e. doing more work. More likely, however, is that the manager, who is not a structural engineer, sees a perfectly good bridge and orders the tanks across anyway, causing the loss of the bridge, the tanks, and the war.
It's not debt. It's just (at best) an incomplete solution or (at worst) a bad solution that fails at the worst possible moment - e.g. database collapses during registration for the largest event of the year.
Ah, but surely, if you build the lightweight solution for $1m, and acknowledge the increased costs of managing the problems that it doesn't solve, then thats fine? Sure, but that's not technical debt either! That is scoping: we (engineers + business) identify a workable solution that provides some business value. And then we do that well. When Cunningham, for example, talks about what to do about debt, e.g. YAGNI, these are all good ideas: scoping. But the term "debt" is incorrect in this case. The term "debt" will only ever get a team in trouble as it indicates that the team is unwilling to confront the actual problem as engineering, but only as a broken metaphor.
The debt there is knowing that "we will spend $500k more doing it this way (interest) but we will benefit in the meantime".
There are other factors too that make the true cost of the debt higher. For instance less flexibility as requirements change. I often find myself saying "this would be an easy change if we had initially built it in the more extensible way, but at the time you didn't need/want for extensibility so now your easy feature is actually a large refactor"
There are plenty of situation, outside software, where doing something simple now results in something more complex later. Nobody calls that debt. MBAs don't call that debt.
What you describe there is actually an ideal conversation, and when presented with that choice, managers can make the right choice for them. What you are describing is scoping, done right.
Almost nobody really uses tech debt as a concept the way it was intended by Ward Cunningham.
It seems mostly to be abused to absolve oneself or others of responsibilities. I or others didn’t make a mistake, it was just “tech debt”. Sounds much more benign like this. Like tech debt is inevitable, unavoidable and current issues could not have been prevented. Yeah, sure.
When I hear people talk about tech debt, my bullshit meter is immediately on high sensitivity.
I fully understand why people don’t want to expose their own mistakes or those of others. But I think it kills trust for those who see right through it.
Having to talk about tech debt instead of the real, actual issues feels emotionally immature. I mean this in a kind way, but I mean it. It results in this whole cottage industry dealing with tech dept, in a similar vein as the Agile industry.
I don’t think all this dressing up of rather clear cut issues as tech debt adds any value and only creates a layer of bullshit work.
I think we need less bullshit and more environments where people can openly admit to and discuss mistakes without immediately being raked over the coals / attacked.
But yes having people with the competence to actually understand the clear cut issues also making the prioritization would be nice.
And maybe we should not let people decide priorities who don’t understand the problems at hand and the trade-off to be made.
Bad code is something that happens, even to good devs. Sometimes you just don't have a clear picture of the problem and cobble something together to try and learn how to approach it. That code will be bad, but it was useful for its purpose.
It only becomes debt when faced with the results of your past mistakes you choose short term gains over long term profit, i.e. you know the code is shit but you keep building on it instead of dedicating time to refactor things because you need a feature yesterday instead of tomorrow.
Often this last decision comes from management, not the engineer that figure out things aren't going well.
It was about building software with the specific goal as to learn about the problem domain and the concepts. It is focused on high-level architecture, understanding and discovering the foundational structure.
And once you gain new insights NOT reworking code to align with new insights would result in ‘real’ technical debt.
There's no such thing as a prototype in practice, unfortunately from my experience. If management sees something half working then they don't want to toss it out.
These days the concept of technical debt has simply expanded to all other situations where this tradeoff occurs, where the knowledge is already partially there but the decision to go with the "prototype" (which is not a prototype because it's already known to be a dead end) is still made.
TL;DR: Tech debt occurs any time you write throwaway code to achieve some short term result, which you don't clean up later (pay the debt). Ward Cunningham's original definition is an instance of this more general definition, his is unnecessarily specific IMO.
It was about building something to gain understanding of the problem space and deliberately taking the time to incorporate new insights into the code. Not doing so would make adding functionality more difficult long-term. Obviously not incorporating such learning can be seen as a short-term objective, but it is more about being actually aware that you need some time incorporating lessons learned or otherwise there are consequences.
Your definition of technical debt is just making trade-offs and that just is that: explaining the actual trade-off and there is no need to talk about technical debt. It’s just a bullshit empty label.
> Obviously not incorporating such learning can be seen as a short-term objective, but it is more about being actually aware that you need some time incorporating lessons learned or otherwise there are consequences.
Why would someone take the time to do a learning activity and then not only ignore the results of said activity but actually deliver the knowingly bad attempt? The important part is why someone would do this, the motive.
There is only one reason to do so: they don't want to spend the time and or resources, or need to spend them somewhere else. That's a trade-off of course, so technical debt always involves making a trade-off, even in Wards definition.
I also vehemently disagree it is an empty label, it describes quite nicely the concern through analogy: paying interest (slowing down development) on debt (throwaway code that was kept). There are other trade-offs that are nothing like this.
However what people did is ignore his actual point and ran with the ‘debt’ analogy and now we call ‘not applying patches for a year’ technical debt whereas this is clearly just bullshit.
And for more serious topic, every trade-off somehow gets called TD. Every deviation from some platonic ideal is TD.
It just is extremely unhelpful for any meaningful discussion.
Also, I think that some decisions are to be made within the development team. It is not for the business to tell them how to develop and keep the code base healthy. Some thing you just do and they are just cost of doing business.
Business wants features and never wants to say, ok do maintenance tasks for a month. I’ve seen it play out, a lack of critical maintenance imposed by product caused days of expensive outages across multiple customers.
Thanks for being my sounding board :)
But the rest of the world has taken the concept of TD and warped it to mean ‘my fuckup with a more neutral label’.
Check definitions when you talk about TD with someone.
At the very least, give it a more specific name.
A Reforge article[0] that was shared here about a month ago[1] suggests these six categories:
- maintenance debt
- developer efficiency debt
- stability debt
- security debt
- technical product debt
- decision debt
Really the concept doesn’t add anything.
Are we talking about mistakes?
Suppose that I build a system that has capacity 1M when the usage is 100 and growing at 10% a month. That system will be inadequate eventually, but it won't be for some time.
Should I have built a system with capacity 10M instead? Would it have been horrible if I'd only built for 100k. (It's unclear why those 10xs are different.) Should I have built a system with "infinite" capacity? Does the answer depend on how much each alternative costs?
If stakeholders are aware of design decisions and related cost + understand the potential trade-off, that’s it.
No need to add concepts like technical debt. And this example would not even be technical debt according to Ward Cunningham’s definition.
While Ward Cunningham may have introduced the term, his definition is obviously not the one that most people use, so it's pretty irrelevant.
However, I feel there is a need for an iterate approuch when programming, and I would very much call not going back fixing stuff accruing technical debt. No matter if the fault is "OK" and part of the natural flow of programming or incompetence.
In practice, however, many developers work in a low trust environment where management is quick to bring out the stick for any (perceived) mistake or error. In that scenario, it's understandable to not want to expose your mistakes.
Defects, tech debt, refactors, whatever you want to call them need to be measured against all other work in the backlog and given as objective cost and risk as possible. Engineers tend to like neat and tidy but there is terrible code in our system that has simply never been changed and still works so we leave it alone.
1) Ignore it and gradually ossify as the code congeals and it becomes harder and harder to make any change at all.
2) Assign a fixed budget of time, e.g. 20%, to refactoring and bugfixing and updating models and "moving to the new version of the language/library which lets us ...".
3) Prioritise the engineering tidy-ups alongside features. This involves (a) being able to credibly estimate the effort and value in the same dimensions as feature work is estimated and prioritised; (b) the people who do the prioritisation believing that engineering-desired work is as necessary as sales-desired or PM-desired work.
That last route requires people's beliefs and abilities to align. The middle route bypasses the entire prioritisation conflict at the cost of sometimes working too much or too little on technical debt. I think that's why you see a lot of fixed budgets for internal work.