Reframing Tech Debt
increment.com
increment.com
> Carving out time to pay down tech debt requires buy-in from leadership.
Not in my experience! I think one can do pretty well a) refusing to put debt in to begin with, and b) for any new work, including necessary cleanup of what you'll be touching in the estimates.
Some years back I had an explicit understanding with a product manager: the engineers would track tech debt and take care of it in small slices over time. He could always ask for details, but absent his curiosity, we'd just leave him out of it, doing a little bit every week so that things always stayed in good enough shape that we would keep our velocity high and our surprises low. Early on he asked a couple of times, but he quickly realized that to him it was entirely boring.
I think it's a mistake to ask non-experts to prioritize things they don't understand. You're not going to ask your execs if it's ok for you to take time to eat or brush your teeth. You equally shouldn't ask them if it's ok to be a responsible professional and do things like writing tests and keeping the code base reasonably clean. That should just be part of the deal; any exceptions should be temporary and carefully negotiated.
> You're not going to ask your execs if it's ok for you to take time to eat or brush your teeth.
I like this! I could add: machine shop workers don't ask if it's ok to clean up the mess after a job.
Exactly right, for the most part if the business has enough trust in engineering to do the best they can, the technical strategy to realise business value is up to engineering, including 'debt'.
The most effective strategy I've seen so far is my current employer. We split our time across the quarter. 60% to core product focuses for the quarter, 20% to ad-hoc requests from the business, and 20% to 'tech improvement'. The latter includes personal growth for new engineers related to our stack or exploring new tech, improving our current tooling, and addressing areas of 'tech debt'. This split is agreed with the business, and the tech lead in a team is responsible for maintaining the balance and has autonomy here too - often we see areas of debt that impact our product focuses so that becomes core for example. It takes trust on all sides but seems to be working well.
My experience has been that, once in a while, even this is not enough. You might just get accused of being a perfectionist or being overly meticulous about things that don't matter, and why can't we just do it the easy way and worry about it later?
Of course, if that does happen, it means that you are in a toxic work environment and you need to leave as soon as you can. But it does happen, and I have experienced it firsthand.
Yes doing shit on top of shit will be paid later, but sometimes, maybe that's fine to pay later while you are scrambling to make money in the first place ?
It takes time for well maintained project to detoriate bad end enough for debt to impact business.
Even when it does , both the biz and new tech leaders are likely to refuse to acknowledge debt is important factor or blame it on you or start a rewrite without understanding what is going wrong to build on top of some shiny new tech everyone wants on their resume.
It's also possible to be both meticulous and flexible. Sometimes when I've gotten that "can we worry about it later" question, I'll tell them yes. If we have a big external deadline or something, I'm happy to treat tech debt like a high-interest credit card. We run up the bill in an emergency, and then as soon as the emergency's over, we pay off the credit card.
If it's in the codebase for a single project, then sure, you can do some refactoring while you work, add more tests, etc. It's not too big of a time sink if you do a little at a time, and you can do it all by yourself.
But if the tech debt is in the architecture, or the API spec, or the authentication layer, or in some other area that impacts how services owned by multiple teams communicate? Now you need coordination with other teams to fix the debt. Everyone else has their own jobs to do, and if leadership doesn't make fixing tech debt a priority, other work will always take precedence.
Coordinating teams does make it difficult though.
I agree that if a mess is particular large, or cleaning it up will have product/schedule impact, sure, the you have to talk with the product people. But the converse it true: if you can find ways to do the necessary technical stuff while still producing sufficient business value, then the product people will never notice.
It wasn't squeezed in only when there was room in the sprint, it may not have even been a formal part of the sprint to begin with.
The PM wasn't trying to fill up the entire sprint with everything they could fit that they wanted to deliver. That's not always the case.
> I think it's a mistake to ask non-experts to prioritize things they don't understand.
It is, but many PMs behave where they have to know what everyone is working on each day (standups) and it has to be what their priorities are as well (via the tickets in the sprint which they plan). This PM you worked with was more open. That's wonderful
Will product managers want to do more than is professionally responsible or even possible? Sure. It's their job to want things. Will some people try to get up in your business inappropriately? Also yes. But the answer to that is to set clear boundaries. Product people make product decisions. Technical people make technical decisions. And technical people need to be ready to answer in business terms why particular choices are optimal for the business. As well as being able to self-run the technical team so that others don't feel obliged to step into a power vacuum.
I think the analogy can work in software development.
Probably works for software development too.
This feels like it's probably dependent on the culture in question. Writing tests, caring about readability and the long term sustainability of the codebase and even managing the technical debt all can be the exceptions themselves in certain teams.
Sure, you can turn around and run for the hills, but for many there, those are the circumstances that they put up with to get food on the table, or work with for different reasons, so those circumstances shouldn't be ignored.
Ergo:
> a) refusing to put debt in to begin with
Management: "Hey, what is this ticket taking so long to complete? Can't you just develop the solution more quickly, without spending so much time on refactoring, planning or whatever it is that you do? Developer X did something similar faster just last week!"
> b) for any new work, including necessary cleanup of what you'll be touching in the estimates
Management: "Why are you suggesting estimates that are 1.5x - 2x higher than the rest of the team? Refactoring? Integration tests? Test data generation? Logging? Do you really think that those are necessary for this many tickets? Don't you want to reconsider your estimate, not to be the outlier? No? Okay, we'll just take the averaged estimate of them all."
Personally, i believe that developers should be trusted more and just care about the long term sustainability of what they're developing beyond their probably employment duration at the company (given how popular job hopping is nowadays), yet sadly that's not been the reality in my experience, probably because of prioritizing functionality and business value now, rather than stability and reliability later.
In the end, developers have to decide if they're professionals or not. Me, I think of myself like a doctor or a lawyer or an engineer. That means I have a code of ethics and professional duties. As an example, the IEEE/ACM software code of ethics is here: https://ethics.acm.org/code-of-ethics/software-engineering-c...
It's also reasonable if people think of themselves like a minion or a servant, where they're going to do what the boss says, no matter how dumb, dangerous, or counterproductive is. In which case, sure, they can excuse themselves from this discussion of how to manage tech debt, because they've decided to put that power in the hands of others.
But if they are professionals, then this indicates a good place to start:
> Why are you suggesting estimates that are 1.5x - 2x higher than the rest of the team?
Any improvement in working conditions is best achieved through collective action. That is a great moment to talk to team members about setting joint standards for estimates. And one doesn't have to approach it from a highfalutin' perspective of ethics. If colleagues lowball estimates out of ignorance or while seeking glory, that means they're just screwing over the rest of the people on their team.
Which leads me to this:
> those are the circumstances that they put up with to get food on the table
This week, sure. Not everybody can just walk out the door this moment. But over the next 5 years? Every single working developer can have a plan to get more autonomy and better working conditions. Including by building up enough of a cash buffer that they aren't one disagreement with the boss away from not being able to put food on the table.
Indeed, I think it's vital that they do. If they instead just bump along at a job with outdated tech and high debt levels, how easy do they think it will be to get jobs later in life? The skills and habits one builds while working in that sort of swamp don't look great in interviews.
I worked in a team that had great culture in this regard. Writing tests, code review, and refactoring were considered an inseparable part of every task; when you estimated the task, you estimated how much this all together would take. The task was considered complete when the comments in code review were fully addressed.
This worked well while we had a manager who was not a developer. He asked for estimates, we gave them, we delivered a product in much higher quality than was typical in the company, he was happy.
Then we got a manager who was also a part-time developer (for another team). And he started asking questions like: "Why did you guys estimate this task to take two days? I could write it in 20 lines of code, why should writing 20 lines of code take two days?" "Well, we also need to consider this, and then we need to write a unit test for this and a unit test for that, and..." "No guys, you don't. Do not overcomplicate things, just do the simplest thing that works." -- I don't know what happened with the project later, because gradually all the original team members have quit.
So, there are ways to address technical debt, and some of them do not require involving the management in too much details, but you still need some minimal level of cooperation from the management; at least on the level of "if the results are consistently ok, I will not needlessly interfere with the details just because I can".
Once you get our of analogies and into actual problems, then you can have a reasonable discussion about future tradeoffs. I find even with non-technical managers, debt/wealth is just not a good analogy for code.
Also, often poor code is just part of a larger process problem. You really need to optimize for the whole project, not just code-level to improve the situation.
Two alternative explanations I've begin to use over the last few years that have helped:
1) Inertia / Inert Areas - the reason this 'debt' is a problem is it means future changes are hard to execute cleanly, and previously. An area becomes inert to change unless an intervention is made. For non-technical stakeholders they most important thing is often that we can keep making changes, so this sets all sorts of loss aversion alarm bells ringing in a way the term 'debt' doesn't - this is where you can start talking about risk, which crosses the divide between engineering and other functions quite well.
2) Product Debt - Yes in some cases code is just badly written, which is an engineering concern, but often the reason changes in Year 5 are hard to make because the domain / reality has moved on, and the assumptions baked into the code written in Year 2, no matter how clean the code, are now just wrong. For example, baked-in assumptions of how business works in Market A as you are expanding into Market's B and C. This isn't the fault of engineering, just the reality of a product that continues to persist - core changes are inevitable. Product leaders therefore need to also get strategic about when it makes sense to rethink previous decisions driven by product needs, and work with engineering leaders to mitigate otherwise you might end up in an inert ball of mud unable to hit goals.
In the most idealised vision of a piece of software, changes will get harder to make in a linear fashion if previous decision are not revisited will new information. In worst case it's more like an exponential graph, so intervention steps will always have to be taken.
As you say, once you're talking the same language, then you get into the details about what strategies to address this inertia / current conditions.
So in a way the beauty of the implementation matter a lot less than its ability to change quickly which sounds obvious said like that.
And you can bet the farm that some dev (or team) was trying to build in some flexibility in to the models which might have added an extra 2 weeks to deliver, which would save months of re-development a year later, and was shot down with 'YAGNI!' or just... "we don't have time!"
For some reason "management" folks get to plan out for the next 4-8-12 quarters, but a dev putting in basic plumbing to accommodate change that will happen 6-12 months down the road is simply rebuffed. "Oh, developers just want to write stuff all the time, they don't understand business", etc, and other nuggets I've heard over the years - grates me to no end.
"How much faster?"
"Dunno"
Plus, at least half of tech debt is about risk not time spent which is even harder to quantify.
For the kind of tech debt you are describing (e.g. making adding a 4th payment system easier) it makes much more sense to wait until it is needed again and to just do it (this offsets the risk of it never being needed at all).
Maybe trying to think about "tech wealth" or "tech capability" might change this. Instead of a ticket to "improve error handling" it's "ensure the system can recover from a DNS failure safely to the DB without manual intervention". This sounds more like a feature, because, well, it is.
I've found most of the places talk a lot about having good quality, but few actually did anything concrete about it. Most changes ended up not making any impact. And this is over the course of a 20 year career.
I'm curious because our team is struggling with tech debt. We are divided in feature teams and I had this idea of maybe simply having a "tech debt team" that, just like feature teams, has sprints achieving tech-debt related goals such as "improve startup times by 10%" or "decouple router module from auth module" or "setup webpack to tree-shake modules" etc.
How do y'all do it?
To that end, keep cleaning as you go. And accept that good development can be like a good batter. Probabilistically good. Not certain.
You want to avoid refactoring if no one else is going to test the change, or if your automated test coverage on the module isn't too good. It's too easy to introduce changes that won't be noticed before deployment.
Chances are most code already works well enough even if the design or code is suboptimal - often the system only changes in certain parts and changes there often. If you can identify the parts of your system that are likely to change due to changes in scale or business requirement, than that is a natural area to focus on in future redesigns or refactoring.
Please don't do this. If it's one team's job to clean up the messes other teams leave, you've created a terrible incentive for other teams around working clean.
I remember one place where each project had a fixed-in-advance schedule and engineers were rotated among projects based on whoever was available when the project officially started. Since people rarely worked in the same area of the code base from project to project, the incentive was to get something working and then GTFO. The code of course got steadily worse. So then the company would compensate with major rewrites of the most garbage-y sections.
The development team does not set the priorities. The team are simply the builders, like a construction team. If you tell a construction team to change what they're building in the middle of building it, your project will take longer and become more expensive, and structurally/functionally will end up worse and worse as you continue to build on something not designed for it. Or if the builders simply decide to skip vital steps to move faster, then of course debt will follow; don't install a vapor barrier in your new home and you will soon find a quite large debt after it's completed!
The only way to prevent tech debt is to have good priorities and not change them mid-way. If you design it well, and have experienced developers, they can built it right the first time with no debt. But if priorities are compromised, debt will inevitably follow.
In terms of the "tech wealth" listed here, what they really mean is "be able to do your job"; have the right tools and processes so you can get something done in the right amount of time. Again, that's priorities. If you don't have some automation or tool, you have two choices: you can buy it, or you can build it. Building it costs more money and takes more time - debt!! If the priority is reduction of short-term cost, the development team can't just plan around that.
Instead of doing that, the key to staying on top of keeping your tech stack in a sane state is to assign ownership of it to a person that knows what they are doing that can effectively counter the pressure from non technical persons in the organization to move faster. It's all about prioritizing correctly in the end and that's not a job you can give to a non technical product manager.
I actually am a CTO that is also responsible for a lot of the day to day product management. So, I get to wear both hats.
My job is not to blindly execute somebody else's vision but to actually create and define the vision and adjust to needs and wants of our business based on what is technically possible with our stack and ensure that that stack is ready to do so. Additionally, I need to stay ahead of this game and a lot of what we do technically is about being ready for what I can see coming next rather than just reacting to current needs. The game here is avoiding over engineering while at the same time spotting and removing technical bottlenecks.
Technical product management is a thing; or rather it should be in almost any team. It's a natural progression for a technical leader to also start paying attention to what they are building and why. I've been interacting with product managers for years and have many product manager friends. Working with them instead of against them is key. Technical product management requires somebody with technical know how as well as a sense of the business and its priority. A non technical PM is the wrong person for this job. If you have such a person guarding technical quality and business alignment, things run much smoother than if you have a lot of people bickering with each other over technical perfection or doing quick fixes.
The important thing is to have a way to show the value of doing that work.
Call it debt, wealth, or whatever. But, measure some KPIs, establish a tech debt/wealth workplan with predicted goals, execute, measure again, and use it as proof. The fight for buy-in to continue incorporating that work in the schedules and plans.
The alternative is creating the work standard of having "20% of your time can go to anything you think needs fixing, no questions asked. Above that, ask first." Just like Code Reviews are no longer seen as skippable (ahem well mostly). That 20% is ALWAYS on, regardless of any deadlines, planned work, be it features, training or peer reviews.
When we use the term tech/technical debt, we are pretending that the real answer isn't "we didn't do it right the first time". The primary reason this phrase even exists is because if we told our superiors the truth then heads might roll. Non-technical people don't really understand what "technical debt" means, but the word debt at least makes some sense to them.
"We need this thing done pronto. How soon can you get it done?"
"Well, sir, it's gonna take at least a month. Maybe 3 weeks if we get cracking at it."
"A month? But we want it ASAP. Why a month?"
"Um, because there's a lot of technical debt we've got to deal with."
"Uh huh... I see. OK, well, carry on then."
walks away
"I think he bought it, Fred."
Then things start getting delayed because developers are paying the "interest" (i.e. regressions due to bad tests) on that original "debt" that was paid to give management what they wanted.
> Development then buys time with tech debt (cutting corners, copy pasting, not writing docs/tests).
> Then things start getting delayed because developers are paying the "interest" (i.e. regressions due to bad tests) on that original "debt" that was paid to give management what they wanted.
The problem with this analogy is, unlike financial debt, said tech debt is often caused by management, and I'm sure you'd agree. Imagine trying to pay down your credit card and Visa tells you to make more purchases or else you may lose your card entirely (and still have to make your monthly payments). Does management view tech debt as something they share or are responsible for? I doubt it.
What the use of "tech debt" actually does is create an artificial sense of legitimacy around all the problems that exist in a failing process. It obfuscates specifics from management, dazzling them with terminology they believe they understand. It allows developers to believe that nothing is truly their fault. It gaslights developers into seeing a counterproductive or even abusive relationship with management as just debt that they own.
It'd be one thing if it actually did work like debt and, yes, I'm sure it does work that way with some teams. I don't think that's the norm.
It’s a way to act like one “meant” to implement a shitty architecture instead of admitting one’s lack of ability and experience.
It’s the mantra of the expert beginner of which there are so many.
You think I’m wrong, but you know I’m right because you can’t define it with any precision no matter how hard you try.
For my clients, when we discuss "tech debt", I give it to them in concrete terms.
"You want Capability A at Month 3, but in order to pull that off, you're going to have to give on Requirement 1 and Expectations Lambda, and it will cost us Y months of implementation in Q4 instead of Y-2 months, will that work for the business?"
"Sure."
"So marked in the meeting notes, and filed under our Tech Debt Jira backlog Feature, we won't move forward with Capability A implementation until you send Sam when you need the technical debt paid off by, and we'll slot it in ahead of time, okay?"
"No problem, we'll get it to you with more detailed requirements on Capability A next week, thanks!"
Sure, when you don't track tech debt it is just sloppy work ("incompetence" is a little harsh, I tend to say the staff involved need more coaching and are possibly more junior than their roles require...bees, honey, and all that). It has served me well to get through to those managers who initially demand the impossible.
No, I was on my way to upvote your comment when I got to
> but you know I’m right because you can’t define it with any precision no matter how hard you try.
It's when somebody decides to optimize for one thing that temporarily mattered to them at the time, at the expense of other things that matter afterwards.
Eg, having that demo next week may truly be existential for the project, but all the foundations you didn't dig and the rebar you skipped putting in to tick that box mean you cannot successfully build on it without going backwards first.
DEV: "We can only do that by cutting corners, reusing code for purposes we shouldn't, and making it not extensible in the future"
PM" "Yeah but we promised higher management so it needs to get done"
some time later...
PM: "Hey can we add feature Y by The Date(tm)"
DEV: "It doesn't fit with the rushed work we did for feature X, we'll need to rewrite quite a bit of code to make those two fit"
PM: "Yes but we promised feature Y by The Date(tm) to this customer, so it needs to get done"
DEV: "Well we can hack something together which does 90% of Y on a full moon and breaks otherwise, I really don't recommend doing this."
PM: "We have no choice"
some time later...
PM: "Why are all features taking so much longer to complete than they used to"
DEV: "Massive technical debt"
PM: "Ah yes, the software industry's way of euphemistically describing their incompetence."
This is an inexperienced dev talking. If s/he was more self-aware, it would be
DEV: I'm not very good at architecture so whatever date you need it by I'm probably going to make a relative mess of it. If you give me more time, at least my rate of introducing mess into the codebase will slow down, so there would be _that_.