Tech debt metaphor maximalism
apenwarr.ca
apenwarr.ca
The end result of this thinking is the feature factory, where a company only ever builds new features, usually to attract new customers. Necessary refactors are called “tech debt” and left to pile up. Yes, this is just another view of bad management, but still, Ward thought that the metaphor afforded it too easily.
He said he wished instead that he’d coined “opportunity,” as in, producing or consuming it. Good practices produce opportunity. Opportunity can then be consumed in order to meet certain short-term goals.
So it flips the baseline. Rather than having a baseline of quality then dipping below it into tech debt, you’d produce opportunity to put you above the baseline. Once you have this opportunity, you consume it to get back to baseline but not below.
I’m not convinced that the concept phrased thus would have the same traction. Still, I love this way of looking at it, like I love much of Ward’s POV on the world.
After seeing a few too many tech debt reduction programs yield even more tech debt than they started with... I simply don't buy retrospective arguments any more. Decide what is acceptable/unacceptable at design time, don't strangle yourself on it after the fact.
I also don't understand what "design time" could mean these days. 30 years ago, when release cycles were 18-36 months, it was a real thing. But now? With modern release cadences, we are always designing the system. Which is great, because the quality of design is limited by our knowledge, and our knowledge increases over time. Setting our standards early on in a project, when we're most ignorant, guarantees we'll be setting the bar wrong.
This seems like an artificial constraint, there are obvious basics such as
- Code should be reviewed by N people
- Testing should cover X core workflows/Y branches etc.
- Dev environments should be creatable in X minutes/hours/days
- The system should perform specific tasks (if you don't know what these are, then coding the system may or may not be the best use of time)
- The system should support Y TPS at Z cost (all systems have a maximum scale, estimate the scale that is required).
- Design should be signed off on by Y people, where Y may be 1 person in a small team.
etc. You can find the right set of constraints for your domain. However a system not doing something it wasn't designed for isn't really tech debt in my opinion, it's just a new use case which needs to be built for.
There is a balance to all things, but I tend to only view tech debt as "work that we agreed that we needed but couldn't fund, and/or intentionally defered."
It really isn't. Again, that's when the least is known. I think a much better point to set a particular standard is when there's some sort of team disagreement that would be solved by setting a standard. At which point you'll have more information than you would have had before starting.
> I've seen more than a few engineers throw the term "tech debt" around as a justification for rewriting everything to their preference.
Sure, and that's bad. But it's bad because there's a dispute over the team definition of "good design" that needs to get resolved by the team. One way to avoid that becoming a massive problem is to regularly set/update design decisions as circumstances demand it. Because sticking with bad or out-of-date choices made before the project started is a good way to create the sort of tension and chaos where "rewrite it all in shiny new framework X" can seem like a good idea.
This probably depends on the size of organization, and the likelihood that the product will survive long enough to require such revision. In my line(s) of work, the odds have been high and it's proven critical to make sure these points are clarified early. If the project is likely to be canned in 6 months anyway... then people can do whatever they want :)
While this stance is fundamentally Anti-Agile, I don't see this as a problem. While Agile's emphasis on rapid iteration is correct - my experience has been that folks forget the importance of testing a meaningful hypothesis or achieving a meaningful milestone in these iterations. In a vast number of domains, it's much easier and more effective to do detailed design work then jump straight into coding. The design process should hopefully land the team on the correct thing to build, how they are going to build it, and why they are building it.
Technical debt is absolutely NOT just "any mess that makes it harder to improve the product", not in its original formulation, and that's what GP comment is saying. Ward says directly, "A lot of bloggers at least have explained the debt metaphor and confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later and thinking that that was the primary source of debt."[0]
Technical debt here, and in the linked article, refers to specific, intentional choices about features and behavior. I've taking to call thing formulation "Classic" or "Traditional" technical debt, to differentiate it from the more widely used definition that includes "any mess". In other words, a mess is not a Technical Debt, or as Andy Cleff says[1], "you don’t have Technical Debt – you have a mess, you have cruft: An unpleasant substance, the result of shoddy construction, redundant, old or improperly written code."
0 http://wiki.c2.com/?WardExplainsDebtMetaphor
1 https://www.andycleff.com/2022/03/technical-debt-what-it-is-...
What you mean is "Technical debt absolutely WASN'T just any mess...But now it is."
More useful to whom?
The reason is that it's pretty much irrelevant afterwards. If I need to work with some code but that will be slower because it needs to be cleaned up, I don't particularly care why it's a mess. I'm not going to indulge in corporate archaeology to figure out exactly what the intent is.
I also believe you're wrong about the linked article. For example: "Bringing that back to tech debt: a simple kind of high-interest short-term debt would be committing code without tests or documentation." Per Cleff, that isn't tech debt. (I suspect Ward would more flexible here.) But to this author and for the common usage, it is.
I'm not sure how relevant this is, but a feature factory is the result of people contracting software development, and measuring it by features; nothing more.
Any other characteristic is a consequence, not a cause.
But the specific format of specifying the software, contract it away, and pay for feature size leads to the specific kind of failure that we call "feature factory".
You don't want to mandate their internal processes, at that point they're not contractors they're your employees and you are essentially taking over their management.
EDIT: Grammar
See the principal agent problem in economics.
Unfortunately subsistence software engineering isn’t a viable profession because the productive capacity of a single engineer is limited. And we occasionally need physical stuff like food/water.
I think a good D&D analogy is your group needs a fighter. Instead of leveling up your decent level 6 fighter to level 7,8 and 9, instead you start multi classing, picking up some droid abilities cause, hey, surely it's cool
Regardless, here is something that describes feature factory in more detail: https://www.productplan.com/glossary/feature-factory/
A huge part of feature factory, worth mentioning, is any lack of checking (let alone measuring), did we build the right thing?
In a feature factory, the developers do not get to know if there is a roadmap or not. It's not of their concern, and the competitive nature of the bids ensures that they won't have time to think about it anyway.
I don't think that this arrangement is even possible for hired developers.
You may end up with things like compatability modes, extra options, and the like. But don't worry, you can clean it up later.
Politicians use words like "justice" or "freedom" to create common ground with little controversy even when they define the term in a way diametrically opposed to their constituents.
I think it can be the same with "tech debt".
I have a friend who figures when he sells his house it will be a tear-down, so he has all kinds of "house tech debt" that he just manages to live with until he literally moves on. it basically means "I don't care about fixing that" or "I'm not going to fix that ever".
I suspect that at the foundational level of this metaphor, the true problem is that some/most people treat financial debt the same, until they're evicted and the car has been repossessed and they've got a warrant out for their failure to show in court.
I want to be wrong about that. Someone tell me I'm wrong.
But I agree your term works a little better since you can have both assets and liabilities and ideally you want to be wealthy not indebted.
i think this is the wrong side of the balance sheet, so in dercuano i put my fonts and css in a 'liabilities' directory instead
Isn't this exactly how people mismanage financial debt? If anything it shows the analogy is a good one.
Kellan Elliot-McCrae has a similar take on technical debt. In his view there is very little technical debt as traditionally defined - an intentional decision to take a shortcut now that we remediate later.
Instead, "tech debt" is typically a label applied to code that makes developers sad because it resists change. Kellan breaks it down further in this article.
https://kellanem.com/notes/towards-an-understanding-of-techn...
That said, I think Ward Cunningham's view is also correct in the context of how debt is viewed today. That said, historically, debt has been regarded as something which was bad for the borrower and only good for the lender.
The distortion likely came about because of reserve bank near-0% interest rate policies enabled by unlimited 'money printing'. Once reserve banks started offering near-0% interest rate loans, private banks found that they could still make big profits by making loans to people on very low margins like 2% to 5% interest rates (since they could themselves borrow upstream at 0%)... But these rates are impossibly low if you think about it. Would any private citizen in their right mind loan their own hard-earned money to other people for 5% interest per year? Let alone while knowing that inflation is almost 5%? Even ignoring inflation which entirely cancels out all the profit, nominally, it barely even covers the risk of the borrower stealing the money and leaving the country... Today, private lending doesn't make sense from the lender's perspective, even when you ignore inflation. Direct lending of one's own money today only makes sense for very risky short term loans with unethically high interest rates (loan sharks).
Anyway, to get back to the point; low interest rates brought on by reserve bank manipulation are unnatural; they made the deal unnaturally good for the borrower (at the expense of the average citizen). Historically, debt was in fact correctly perceived as dangerous for the borrower.
> That feels unexpected. If we're doing so well, why would we want to take on more tech debt? But think of it this way: if your product (thus company) are really growing that fast, you will have more people to pay down the tech debt next year than you do now.
The only thing I think I'd add to it is that like actual debt, a lot of people do not behave sensibly with tech debt. One of my big tests in any new job is the first time a product manager wants to put something on credit. E.g., "We have to add feature X right now so we can land a big client. Can we cut corners to get this out the door? We'll sort out the mess later."
Despite things like this making my eyelid twitch, I'll always tell a new product manager yes the first time they ask this. But then I watch carefully to see if they're conscientious about paying down the debt. Some of them are great about it, prioritizing the deferred work. Others, though, always have a next emergency. But in my view if it's always an emergency it never is, and so I never again trust them with the tech-debt credit card.
For sure, if it's a small client or far from a sure thing, then yeah it's bad. Like anything, a judgement call about where the cost benefit ratio lies.
In this case obviously the PM does not have to deal with the fallout, and then the next guy says, "I know this code base is tough, it wasn't by doing, please just try hard and get this feature out"
The other big things I watch for is how much is shipping on time valued, is that the only metric of success? Further, when ROI is calculated, is the return ever actually factored in?
Also it depends a lot on what stage your project is in. If you have zero customers or money coming in, your focus should be on creating a working product and getting customers. That means zero tests for now, just a working product. It will also more quickly validate if you’re working on the right thing or not, or if you need to pivot.
One problem I see is when engineers want to solve the tech debt problem before even proving the product should exist. No, first establish the demand, get real cash flow, then start focusing on tech debt.
If you are an entrepreneur then tech debt is literally the last of your concerns. If you grow enough where it is then it’s a good problem to have - the biggest problem is typically not enough customers, not too much tech debt.
On the other hand if you are working at Google, and know for a fact thousands or millions of people will use your code, then you should “lock it down” right at the beginning, have all the testing upfront, good documentation, etc.
Alright, little finance whiz, let's break it down for you. Imagine you have some money, and you can use it to buy something called an "option." An option is like a special ticket that gives you the right to do something later, but you don't have to do it if you don't want to.
Now, in the quote, instead of money and options, we're talking about computer code. When people write code, sometimes they write it in a way that's not very good. We call this "bad code." It's like having a sloppy and messy way of writing instructions for a computer.
But here's the clever part: good code can be like a special kind of option for the future. It means you have the choice to make things better later on. It's like having a ticket that allows you to improve your code and fix any problems.
Now, when code is really bad, it's like not using that special ticket. It's like not taking advantage of the chance to make things better. So, when we say bad code is an "unhedged call option," it means you're not using that ticket and missing out on the opportunity to make things right. It's like not taking the option and potentially causing more problems down the road.
a call option is specifically an option to buy a security from someone at a prearranged 'strike' price
shoddy work is not, metaphorically, an unhedged call option you bought. buying a call option costs money up front; shoddy work saves money up front
it's an unhedged call option you wrote to sell on the market. that means someone else can exercise that option in the future, obligating you to sell them an asset of yours at the strike price. they will only do this if the asset is worth more than the strike price, so it costs you money
moreover there is no limit on how much money it can cost you. say you wrote an unhedged call option with a $10 strike price and sold it for 25¢ or whatever. if the underlying security now costs $20, you lose $10 if the option is exercised. if the underlying security now costs $100, you lose $90. if it costs $1000, you lose $990. there's no limit to how much money you can lose writing a single option this way
and if it's an 'american style option' you get no warning; this can happen at any time
hope you invested those 25¢ well
also, please don't ever again post an explanation chatgpt gave you on here unless you've verified that it's correct with a reasonably reliable source
Thank you for demonstrating once again that ChatGPT is comprehensively useless!
1. you take on debt to get hold of some cash
2. at some point, you ought to repay the debt by giving up the same amount of cash
3. until you do, you have to spend a steady trickle of cash servicing the debt
In the technical debt metaphor, cash stands for development effort. But:
1. taking on technical debt doesn't give you a load of development effort out of nowhere (i suppose you could say that shipping what should be a three week feature in two weeks is like getting a week of effort from nowhere, but that seems dubious to me - you haven't shipped the same feature!)
2. repaying technical debt generally takes more work than it saved, because by the time you repay it, it's infected other code that was written afterwards BUT sometimes you end up being able to delete the whole thing, because the feature was retired or fundamentally rewritten, in which case you don't need to repay it at all!
3. sometimes, existing technical debt requires no extra work at all (if it's in a part of the codebase you aren't touching), but if it does, it's not a constant amount, it's proportional to the amount of work you are doing on that code
So it doesn't seem like a brilliant metaphor, really.
Perhaps it's more like cleaning your bike chain after a ride through the mud. If you don't do it, it saves you some time today. But every time you ride your bike again, it's more effort to pedal, and it's wearing your chain out faster.
On an accounting sheet I can see my interest rate and how much I'm paying monthly and make decisions. With technical debt, it's all about people time, and people time is a sunk cost to your finance department.
With tech debt, you can ask several engineers and get very different answers about how much tech debt there is on a particular project. It's not well defined, subject to interpretation, and so doesn't fit well with management wanting quantify the objective value of doing something. I think this is the real reason it's often disregarded.
It requires a degree of trust from management to the engineers to properly address, and that trust can be abused. All sides are aware of this and I'm sure I'm not the only one that has used tech debt as an excuse for some minor padding here and there.
DTI is total debt servicing cost per unit time / income per unit time. The numerator and denominator do not have mismatched units, like the author claims.
What's more, the author's proposed replacement measure--an interest-expense-to-income ratio--is quite silly. By only considering interest expense rather than the total cost to service debts, the measure becomes worthless as a means of estimating the capacity to take on additional debt. If you were to have debts that, despite being interest-free, were large enough for payments toward principle alone to completely consume your income, then you have zero capacity to take on additional debt.
The author also proposed that a bank/lender is to a borrower managing financial debt what a CEO is to an engineering team managing technical debt. They say it without conviction, admittedly, but it is another quite absurd notion. Per the author's metaphor, a lender is incentivized to maximize interest income over the lifetime of a given debt (ignoring rising-rate environments); put differently, prepayment is to a lender's detriment. CEOs very much benefit from prepayment of technical debt. It's a very poor comparison.
At my last job, we were refactoring a bunch of Angular 1.x code for a variety of reasons. It took way way more time than anyone was anticipating, and it was tech debt that had low interest and probably shouldn't have been paid off much earlier than it was. People had chipped away at some components gradually over the years, much like mortgage payments, but after Angular 1.x fully reached EOL, and (imo) we entered a new level of efficiency with build tools, that interest rate started increasing.
However, we should not stretch the analogy. I think the author goes way too far. Talking about interest rates, debt ceilings, and ratios doesn’t make sense.
Besides, I'll comment on some of the financial claims:
Debt doesn’t necessarily need to be payed back. I guess most debt is never payed back because there’s are useful projects behind it. If some software is generating millions of profit, should we then pay back all its debt? Maybe it sufficient to just maintain the software a bit a go home early.
Calling a family stupid because they take a loan to go to Disneyland is a very narrow view. Maybe the utility is much higher than the said interest because the family needs a couple of days together (for whatever reasons the economics professor likes to ignore).
> CFOs can calculate that. Engineers don't like to.
Engineers don't get to, in many cases. It's often the managers steering the ship at full pelt.
This kind of language - while sounding deceptively smart - does not help in any decision making because the terms are neither measurable nor actionable... Just meaningless.
What we need are answers to questions like: how do we measure maintainability of software systems? How do we compare solutions and select them? How do we decide on what investment in making the software better is worth pursuing?
Is "tech debt" even something that can be measured and actioned upon? How do we know if our changes to the software made it bigger or smaller? How do we know its impact on the bottom line?
Everybody knows in the whispers of their soul that an empirically mapped out programming culture, is a dead one.
I can see how to tie together all the publicly available knowledge on programming into truthfully-easy to understand education material.
People like Casey Muratori are selling the game of programming out from the bottom up. (and good for him).
Why do it when we have to sell our skills today to put bread on the table?
But programming is more like product design or urban planning than structural engineering or construction.
There's no formula for maintainability or technical debt in the same way that there's no formula for the best town design.
That doesn't mean that all towns are equal, or that we can't say anything about urban planning. It just means you can rarely make concrete rules about it.
Quantification is the basis of scientific process. Are you claiming programming or software systems cannot be a subject of scientific process?
That would be a really bold and blatantly false statement.
But if you mean "technical debt" is not measurable then we are in complete agreement and what I am saying is that if that's the case then talking about "technical debt" is useless and we should try to find other terms which would be more useful to describe software systems and allow us to base our decisions on more than feelings and intuitions.
Talking about "technical debt" of software system is no more useful than talking about its scent.
I'm saying it's extremely difficult and expensive to answer most of the questions that you want to know through scientific experiment. Usually the best you can do is experience.
That's not a bold thing to say. There are plenty of areas that are really hard to study through experiments, but we can still learn things through experience. The whole field of psychology is just on the edge of being amenable to experiment, but it's not like we know nothing about human psychology.
Management is another field where it's more or less impossible to do scientific experiments to determine what makes the best manager, but again that doesn't mean we know nothing about management.
Very occasionally there are great scientific studies of software - e.g. that paper where they found that Typescript/Flow prevent ~15% of bugs in Javascript software. But most are along the lines of "we took some undergrads, gave them a tiny task, and measured how long it took them" which has no correlation with best practices for professional long-term projects - maybe even negative correlation. For example I suspect it would tell you that adding any comments to code is a bad idea!
> Talking about "technical debt" of software system is no more useful than talking about its scent.
Why do you think talking about its scent is not useful?
And yet in both these cases we at least _try_ to come up with quantifiable results of our observations. The point is that without them we are not really capable of coming up with any meaningful conclusions.
> Very occasionally there are great scientific studies of software
For me it seems the body of software available to analyse empirically is large enough now to start doing it and it is happening. What's important is that none of the empirical studies I know measure anything named "technical debt".
>> Talking about "technical debt" of software system is no more useful than talking about its scent.
> Why do you think talking about its scent is not useful?
Because it does not provide any meaningful insight into what should or should not be done to improve it.
We do in software development too. I said that twice already. It's just almost impossible to actually find out concrete rules that way.
> without them we are not really capable of coming up with any meaningful conclusions.
Nonsense. Scientific experiment is NOT the only way to learn about the world. It is the best way where practical, but it simply isn't always practical.
Think about something like UBI. Using scientific experiments to determine if UBI works you would have to take N countries, have half of them implement UBI for 100 years, and half not. Good luck.
> Because it does not provide any meaningful insight into what should or should not be done to improve it.
It does. Code smell is usually used to describe code that is suspicious or uses practices that are known to be highly unreliable (e.g. regex). The thing to do is to either rewrite it in a more robust way or if you can't do that for some reason, to add a lot of testing.
I think this sentence illustrates very well the root of our disagreement.
Saying "are well known to be highly unreliable" without any backing _facts_ is the sad state of affairs in IT industry.
I am keen to see a research done with scientific rigour that would show that regex is highly unreliable.
Like there are many ways to finance your business, there are many ways to make the same product, from coding it from scratch yourself (which ranges from “from quick and dirty” to “using formal methods”) to customizing a open-source foundation, or even a third party's white-label product. Which one is better completely depends on the situation (like with equity vs debt).
[1]:in finance, “liability” isn't “bad”, debt is a liability, but so is equity, and even profit is a liability!
True technical debt analogies are more like introducing something that increases build time on a large team since this is paid continuously.
We just need to accept that in software, sometimes stuff needs to be done that doesn’t have an obvious end user benefit in order to keep the code base viable.
Have any engineers successfully fought against the urge to pay off debt quickly?
Can't help but pay off my house faster than the mortgage...
In tech debt though, that is split into two groups:
- "the business" concretely reap the benefit of accruing tech debt (release faster = more profit), only abstractly feeling the consequences (developers are maybe slower and maybe more miserable than they should be)
- developers suffer the consequences directly (working with badly maintained code that you don't have leeway to resolve is soul crushing), while only abstractly reaping any benefit[1] (maybe the business profits more which maybe gives them more leeway for maybe a better pay rise than they would have otherwise gotten)
The concrete feelings are obvious, the abstract ones are all arguable because you don't live the opposite (maybe this feature would have always taken this long, maybe the company would have always been able to afford that pay rise), and so are easy to ignore.
So then, unlike financial debt, the decision whether or not to accrue tech debt is ugly tension between two groups who don't really understand or have empathy for each others motivation. The business would always prefer you go as fast as you can, and developers would always prefer they work with code that isn't miserable to work with.
Attempts to solve this are pretty mixed in my experience, and generally the business wins. Which is I think, where most people's negative feelings on the phrase "tech debt" comes from.
[1] the obvious counterpoint here is startups and other jobs where you get stock options etc. I have only worked in that scenario once, and my company was a subsidiary of a much larger one, so our ability to move the needle on stock value was pretty negligible. It may be in scenarios where you really can make a difference tech debt decisions are more harmonious. Interested to hear people's experiences.
Every time some feature is added, it's like putting some weights into the backpack of >>every<< developer that will interact with that part of the system in future. Some actions add more, others less, but you have to be careful. Too much weights will slow down the team/company to a point where no meaningful work can be done anymore. To reach a goal you have to add tech debt, but some feature add way more than others. Ideally tech debt is accumulated in isolated areas that do not interfere with other parts of the system.
Ehhhhhhhh too much of that return goes straight to Consumption instead of your bank account. If we’re going for Max Returns, buy the biggest home you can… and rent it out as you live in a shoebox. ;)
(Or buy smaller homes, to diversify.)
1. The "Technical" word pigeonholes responsibility to the "technical" people. This creates all manner of problems (management, strategy, resource allocation etc.). It ought to be seen as "Software" debt, because the whole software company owns it.
2. Software debt packages risk. We need better mental models of that risk.
- This perception is commonly muddied by personal bias---how we think of finances and debt in our daily lives.
- However, it requires thinking like a strategist and/or economist (much more abstract, nuanced terms). The trouble is that the "debt" is rooted in complexity, which, like rocket mass, is inevitable. And we abhor it because solving for complexity is like solving for rocket mass; requires recursive tree-networked thinking.
- The debt is also inevitable because we want some complexity (rocket mass). It's just that it needs be the exact kind we need, which choice-making is difficult.
- This sort of complexity-debt always compounds and each decision to add or remove items from any part of the system changes the principal and the rate of interest and the repayment terms.
- It is layered because software parts compose into software "stacks" and hierarchies, and each part mutates/evolves up and down the towers.
- It is networked. Because software itself is networked. Small local changes can turn into literal chain reactions of events. The meltdown of Knight Capital stands out starkly as an example of unmitigated, un-hedged software debt.
Ultimately, the "debt" part of "software debt" is better seen as if it were a complex opaque financial derivative. Unchecked creation of software debt is exactly analogous to how the 2008 financial crisis came to be. It's all composed of "simple" debt.
Much as I dislike all this doom-speak, I have to acknowledge it is material, and to side with Andy Grove. Only the paranoid survive.
The only real hedge we have is the creativity and intelligence of our people.
More at my blog post here: https://www.evalapply.org/posts/software-debt/
People who develop and manage the code, most likely do not own the debt. Code (and technical debt) belongs to stock holders. And there is a huge separation from developers and management.