A Taxonomy of Tech Debt (2018)
technology.riotgames.com
technology.riotgames.com
1) time to design it 2) knowledge of exactly what it needs to do today and in a year
Sometimes you're missing both.
In which case I think you can prevent contagion from being too terrible by enforcing smaller modules and single responsibility in a compositional way. That doesn't require as much knowledge of the future or time, but just requires you to avoid high-surface-area interfaces that end up with lots of behavioral variants controlled via parameters in a nesting-doll style. Instead, move your config/parsing/behavioral decisions to the edges of your logic instead of letting them seep into all your underlying models too.
I would classify that as thoughtful interface design.
Do good interfaces take more time than bad interfaces to write? Does adding more time really make interfaces better? I find that engineering quality (of which interface design is one facet) is largely a function of talent and experience. Time doesn't usually play a factor. Writing good code takes the same amount of time as writing good code, for the most part.
Much of the system can be complete like this with forethought. The pieces that cannot can be factored out to the edges.
The majority of devs generally are happy to tack on their features and PRs to whatever random scaffolding they can, without regard or awareness for how their individual component fits into the larger system, or how it may be extended. And to be honest it’s not necessarily a bad thing, because they do need to get work done, and merging PRs shouldn’t be reserved for the enlightened.
I guess I’m just pessimistic. The reason we don’t see perfect software is because we are not capable of producing it. At a certain point it all becomes spaghetti. If you work with software that isn’t spaghetti, it’s only because the people who care about it not becoming spaghetti haven’t left yet. This is good, but eventually they will leave, standards will decline, and you will become one with the pasta.
Forethought is only possible if people tell you the requirements precisely correctly upfront. Real systems design is you get 90% built and someone drops a hard requirement that's also a layering violation on you.
Math arises from first principles, human behavior does not.
The best defense against this that I've found is to ensure, as much as possible, that interfaces can be replaced. The single responsibility and interface segregation principles can help here. Using small, focused interfaces and letting modules implement more than one of them makes it easier to use the strangler pattern to replace interfaces that no longer work well with new and improved ones.
Also avoid temporal coupling as much as is feasible. Unnecessary statefulness is the easiest way to make this sort of thing harder than it needs to be.
> 2. . To design a spacecraft right takes an infinite amount of effort. This is why it's a good idea to design them to operate when some things are wrong .
> 3. Design is an iterative process. The necessary number of iterations is one more than the number you have currently done. This is true at any point in time.
> 4. Your best design efforts will inevitably wind up being useless in the final design. Learn to live with the disappointment.
Also 9 10, 11, 12, 13, 14 and a bunch of the others apply too.
An excellent interface will eventually be deformed beyond recognition chasing the architectural dragon; a well-crafted library will outlive the project.
That won't happen. Why toy around with your ticket database like that? Just close it to WONTFIX.
In computers, you have x86 being the poster child of ostensibly suboptimal interfaces.
Would be like complaining that AC is being superseded by DC and how this is proof of an early choice locking us into a bad choice. But it ignores all of the progress made in the interim. And the odd reality that enough effort can migrate anything. It just takes a lot of effort. And we are often quite willing to throw effort at things.
Although to be fair, we don't have any EMs who were promoted from within. We have a bad habit of hiring managers from outside, as nobody internally really wants to stop doing engineering (myself included).
Founder's debt.
This was debt that was created by the founders to get the fast, good value tech out the door. Low hanging fruit that ends up being the foundation of the whole shebang.
The founding documents of many countries fall into this category lol (but not USA! USA! USA!)
Macgyver debt and foundational debt come closest, but neither quite outline this phenomenon.
As always, I have a philosophical nit to pick: the “three axes” introduced at the top are just “Return” and “Investment” from good ol’ RoI, with a subcategory added for a particular type of forward-looking/conditional Return. I’m guessing this decision has worked in practice and I don’t expect video game development practices to be absolutely scientifically sound, but some extra philosophical certainty never hurts!
A Taxonomy of Technical Debt - https://news.ycombinator.com/item?id=16810092 - April 2018 (113 comments)
also this bit:
A Taxonomy of Tech Debt (2018) - https://news.ycombinator.com/item?id=39782923 - March 2024 (1 comment)
One of the best descriptions I’ve encountered.
As in all debt, there’s a “threshold” that should be applied, at the time the debt is incurred, which balances the immediate needs, against the future costs. I feel that most people (not just developers) amplify the immediate, and deprecate the future costs.
For myself, I have an almost pathological aversion to debt, of any kind. I will spend an extra day, factoring out stuff that might be useful, in the future. I seem to be right, about 50% of the time. That said, every time I do something like that, I reinforce habit, which accelerates my basic workflow.
Perhaps if 24 minion instances constitute an actual problem (rather than just inelegance) their example for it is actually foundational debt related having a "minion" being the simplest primitive that would do the job when maybe something lighter could have existed.
Encouraging devs to have their changes include all modules, even those that are old and mature and don't need to be touched, is a good way of ensuring this doesn't build up to where it becomes a problem.
It's a tool, but a powerful and dangerous tool, and if you don't acknowledge you're using it and respect it, it'll hurt you. Or it'll hurt someone who accepts the grenade from you. Just like real debt.
When you have a proper backlog of tickets, including tech debt tickets, the team will eventually fix the tech debt when there are not enough feature tickets to exhaust capacity.
techdebt can even pop up in unorganized slowmo opensource software.
That incident really changed my perspective on people who talk about how tech debt is bad. Some of them will roll up their sleeves, but some just want to look high minded without putting in the effort.
I have yet to visit this misterious universe you describe.
The trick is to have 1 backlog. Tech debt and features live on the same list and it is up to the PM to prioritize. Engineering’s job is to argue cost.
Good PMs will prioritize relevant tech debt or pull it in with feature work in the same area. They understand the tradeoff of go slow to go fast. They also understand when tech debt will never become relevant (because the feature is getting nixed, or hasn’t shown desired impact yet, or because the cost of interest is waaaay lower than the cost of paying it off in many cases).
This only works when engineers have the discipline to look stinky awful code in the eye and say “not today” and stay within agreed timeboxes. You blow this estimate once or twice, get the PM in hot water with leadership, and you’ve lost the trust.
For teams that don't have a good PM, you also need a tech champion. Failing that, engineers need to inflate estimates and do tech work under other stories. Then everything becomes less predictable and teams never develop trust.
Yes. And to add some nuance, you need a [trusted] engineer who can say “This will take 3 weeks because of tech debt items A, B, C. We can fix those in 1 week and then take 1 week to implement this. How would you like to proceed?”
Any decent PM will take the 2 week option that also cleans up the codebase.
But if fixing the tech debt would take 3 weeks and then another 2 weeks to build the feature, then any decent PM will take the option that doesn’t fix tech debt unless there’s a bunch more stuff coming in this area in which case taking 3 weeks to fix stuff is totally worth it.
Their job is to make those tradeoffs. Our job is to highlight the tradeoffs they’re making so they can make informed decisions.
> Our job is to highlight the tradeoffs they’re making so they can make informed decisions.
This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.
If you have shit engineers and great PMs, the best outcome is likely to shift decision making to PMs. If you have great engineers and shit PMs, decision making should shift towards engineers.
If they are both equivalently shit or great, it should be a balance. I believe this is the most likely scenario. I believe that balance is thrown out the window if engineers "highlight the tradeoffs" while the actual decision making is lies with the PMs.
How to actually achieve balance is extremely idiomatic to the team and organization. It's hard to get people to have adult, non-confrontational discussions about this sort of thing, however. Too many people will treat it as a negotiation.
I think of it more as a partnership.
If I’m in charge of getting groceries and you’re in charge of budgets, we need to have an informed discussion on what exactly is our budget and what food we need so we don’t starve. Sure I could blow the whole budget on steak and I might even love eating nothing but steak for 3 days, but eventually some carbs would be nice. Likewise neither of us will be happy if I go max stingy and buy nothing but bags of rice for the week.
The reason I think PMs should make the final call is not that engineers are subordinate, it’s that PMs are accountable. (RACI – responsible, accountable, consulted, informed). The person whose ass is on the line makes the call.
Usually when I ask engineers if they want to be accountable for making the call (and its outcome), things get real quiet real fast :)
From what I've seen, accountability doesn't mean much. Could be the places I've worked. Poor PMs get promoted despite running projects into the ground, good engineers get held back despite pushing through adverse project plans, vice versa.
I've experienced something like this, but only on a project that mostly had the original team that built it (including me) still working on it. We were able to keep things in check, and in the above case would just do it that way without really asking.
On many other projects I've been involved in, there's years of tech debt that has accumulated: the typical retrospectively incorrect design decision, followed by layers and layers of band-aids, each time making the real fix more complicated and a bigger scope.
These things undoubtedly increase the cost of everything else, but it's really hard to articulate. The fixes take weeks, the break-even won't come until months later, the long-term team members are a mix of skeptical and defensive of their work (eg: don't want to do the real fix). In some cases, there's a war story "we heard that about x, but that caused so many bugs we had to revert and abandon it, why is this going to be different?"
Any tips for anyone working in this environment?
The problems come from the calls where personal incentives are not aligned. A typical example - the team builds a feature hidden by a feature toggle which is, after a period of A/B testing, enabled globally on the product.
The existence of the feature toggle raises the complexity of the code - let's say it's used in 10 different places, each of those double the amount of possible code paths. Removing it may be a question of a couple of hours of work and is very clearly work paying for itself in the long term, but PM will not schedule this work, because there's no immediate upside for them personally and the cost of keeping the toggle in code is a long term one, spread over the whole organization.
In other words, PM is more likely to get a bonus by slashing work on such tech debt items (and thus them personally delivering the features faster) rather than punished for keeping the toggles/complexity behind.
That's part of the role of a Technical Program Manager. The Eng Manager, Product Manager, and TPM should form a holy trinity of mutual support, filling in for each other's gaps. When that happens, you get much better odd of having a high performing team.
Source: I've been both an engineering manager and a TPM. Never the PM, though.
That’s a lot of words to say “more features lol” which is basically what every PM I’ve worked with has only wanted.
Most people don’t want to work on it. That’s why there is so much. Generating it is like eating candy. It’s unhealthy but you just want to have something sweet right now and the bowl is in reach…
Most co-workers I’ve had would have _loved_ to fix the shortcuts and hacks that were done to meet deadlines, but were never given the time. “Refactor while you do new features” works sometimes, but doesn’t work on anything larger scale - e.g. if your overall architecture is collapsing under its own weight, it’s hard to “sneak in” the sort of major work you need to do to fix it.
Yeah, this is 100% correct. I comically left Riot after ~6 months for this exact reason. Obviously it's a large company with many different flavors of teams, and it sounds like this team maybe has gotten it together, but by in large most haven't.
While I was there I was working on some of their core games tooling and felt uneasy about my day-to-day. My teams tech debt was quite literally owning them. Constantly missing sprint scopes, spending countless hours arguing and debating about trivial stuff, it was all a mess. They ended up laying off a number of people from that team in a pretty shifty manner so maybe things have gotten better since then.
This is dev culture not agile culture.
This puts you at grave risk of redundancies.