Is “Investing in improving structure” better than “Paying off technical debt”?

I think this is related to how badly things go wrong if there is a fuckup. You really do not want to push the engineers responsible for determining your $500M office building will stay standing.

It’s because there are literally infinite ways to solve similar problems in software. There is no physical assets to measure the differences making very hard as relatively speaking it’s still a very young domain. I still go back to my freshmen year in college when in my dorm we had a mechanic, chemical and material science engineers. Each of them complained because they were not senior enough they could not get time in the expensive lab to run their end of year projects... it was at that moment they I looked at my crappy dell laptop and realized I had access to the most advanced systems in my field computer science and it was just sitting here in my dorm... I mean a few years later a guy on similar crappy laptop or desktop created Facebook... and yet there are so many options to build Facebook using php, c#, c++, Java, python etc... maybe to you one of those ways is inferior or less good then the other. Maybe using cloud servers or physical would be better?? Maybe a class or a function... css or Scss... the space is infinite

Speaking collectively, given how much we fight against standardization of our profession, we should be distrusted, IMHO.

I think it’s about perception. Broadly speaking, professions like Doctors and Lawyers are seen as prestigious and important, and so they are allowed to manage themselves.

I think software engineers tend to overestimate how prestigious and valuable people outside of tech think software engineers are.

I work in computational biology, and even in a fairly tech related field, you wouldn’t believe how many people think software engineering is just glorified IT. I’ve had a lot of conversations where the crux of it is “why should we pay them so much money, it’s the computers that are doing the work. Or, on building a distributed computation engine to run for 2 years straight without failing: “it’s all just grunt work we’ll just get an undergrad to do it”.

People in other professions have similar issues. Some 'smart' folks feel as if they should be able to understand everything. It probably often works for them, after all, how did they become 'smart'? But, it's a fine line between genuine interest and distrust per se. Try to assume good intent, consider it interest and grow your ability to explain. At some point you might have to call it as a poor use of time and move on, but if you can use the time to grow on both sides of the discussion - bonus.

> if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

Physicians enjoy a similar problem. Everyone goes to Dr. Google an then brings their misunderstandings to the clinic. Anything goes wrong, it couldn't possibly have been the fact that the patient was a brittle diabetic with hypertension. It must be the care they received in the hospital after they had a heart attack while driving and ran into a tree. Nor is anyone else in the hospital possibly making mistakes. Clearly it's the board-certified physician who doesn't know what they're doing.

From my experience, managers tend to start to distrusts software engineers after the first complete system rewrite, that was supposed to solve the big problems, actually didn’t solve them and brought new ones instead.

The issue is neither has crossed the bridge. The non-technical manager can’t understand the risks, and the engineers can’t quantity it in the dollars and cents that CFOs understand. Salespeople are good at dollars and cents. So are some lawyers. But they get overruled all the time by execs.

“I don’t care how you do it, increase sales by 20% in Q4” creates the sales and marketing version of tech debt. (Channel stuffing, excess discounting, etc)

I distrust my fellow software engineers when they say they need time to pay tech debt.

There were many times where I refactored code that never received any additional updates or to have the feature removed. I wasted a lot of time without any real gain for it.

There are real costs to paying off tech debt and it is easy for engineers to spend their time on low value refactoring.

Software engineers are not always good at prioritizing the business value side of their work so they aren't trusted there.

If you can show you can prioritize business value then it is easier to convince management to give you two weeks to clean up tech debt.

> The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

Yeah.

I think, sadly, this is mostly on software developers though. Look at all the pop culture stereotypes of programmers: whether those stereotypes are true or not, that is what culture thinks of us. Would you take seriously what a big-bang theory character would say? The problem is, I don't see most software engineers valuing social skills to better relate to other professions. Instead, there's a lot of whining about being misunderstood. And we are misunderstood! But man we're not helping it at all.

> Do people in other professions have the same issues?

There are very few professions that have as much leeway as we do to begin with.

In a sense, technical debt is only a big problem for large monolithic codebases. Software that is primarily composed of small manageable components means that technical debt doesn't matter as much.

When each component is roughly one week's worth of code, it becomes easy to delete a component with a model that no longer fits the problem domain, and rewrite a new component with a new more appropriate solution.

"If you can rewrite a piece of code in a week, you can understand it in a week." - Greg Young

https://maximilianhoffmann.com/talks/programming/the-art-of-...

I think it's a bit different in software because it's fairly easy to change things and update them.

As for the distrust, generally there is can be a lot of distrust in things you don't fully understand. Legal I think is a great example. There are a lot of contracts that screw people over and legalese is so hard to read, I have a hard time trusting what is going on. But one big difference is it is very difficult to go back and change legal terms. I was changing an LLC to a c-corp in order to receive investment and it took 3 months to fix the "legal debt" I had incurred by starting with an LLC with outside parties who had a say in what the new terms were. And it's hard to say whether we did all the right moves in that transition because I'm not a legal expert. Just as some people doubt the decisions I make in software.

I think one of the problems is that code is a black box to non-programmers. If I tell you that your car needs $2000 worth of work in order to avoid a $10K problem down the line, how do you evaluate that? Keep in mind that as far as you can tell the car is performing exactly as you expect at the moment.

Another problem is that especially in large organizations there are people who are literally trying to scam their way into money. You always have people aggrandizing their work, or belittling others. Especially as companies become profitable, they attract people who are only out for their cut. There are people who do nothing but attend meetings and try to scoop money into their pockets. This is the norm.

Now marry the two issues and you can see the result: someone says they have to do "special work" that will delay the project and that they are actually saving everybody else's asses. Nobody that isn't involved in doing the work can evaluate the claim. Are they just avoiding work/blame? How do you know?

I find that usually in organisations I work with, there is trust up to the D-level manager (i.e. those who directly manager developers), but there is often very little trust from other parts of the organisation (sales, upper management, or sometimes even program managers).

The best way to engender trust is to work as tranparently as possible. In the best of worlds you would deliver something every day and it would be obvious what's in the queue. Things would be in the queue for a very short time (say 2 weeks max). Work like refactoring, writing tests, improving build systems, etc would be just things you do every day and nobody would ever mention it. Of course, not everybody lives with an ideal situation, but I think most teams should strive to move their situation as close to this as they can (there are definite exceptions, though).

Here's what normal people see and hear when you enter the room: Nerd opens mouth: "Blablabla quibberish, you're all idiots, hear me express your problem way overcomplicated. Also, I need more budget and cannot promise to actually deliver a solution anytime soon because of more quibberish."

If you want to be trusted start acting like normal people. Stop talking in technical quibberish. Stop behaving like a nerd. Shape up. Get some social interaction with normal people. Stop complaining all day long and start delivering what was asked on time and on budget.

Yes, business stakeholders often mistrust software and IT people, because relative to other industries we do a poor job of meeting our commitments.

McKinsey did a study recently which found that large IT projects ran on average 45 percent over budget and delivered 56 percent less value than expected [1].

An executive is interacting with people from multiple industries or business functions and this is on the high side. It's much worse than a function like accounting, and it's also worse than many industries with bad reputations, like construction. So over time they are going to come to regard estimates from the tech guys with more suspicion than most. To the extent that other industries have this problem, yes you will find similar trust issues.

If you are a PM/product manager then the best way to handle technical debt is to focus on the benefits that come from paying it down. You are not scheduling time for technical debt, you are scheduling time to improve reliability, security, scalability, etc. and from day one you are lobbying for more time to do every feature so that it's done right in the first place. Yes, sometimes one of these things has to go wrong before the business people start listening, but if you were calling it out from day one they will respect your input that much more.

If you are a developer it's important to understand what work has business value and what doesn't, and make or inform decisions appropriately. Frankly there are developers out there whose attitude amounts to "Your job is to pay my salary, my job is to make whatever technical decisions I want without oversight, and if we don't deliver on time/on budget it's not my problem."

Not all tech debt is worth paying down, the decision generally should be made based on the "interest rate" of the debt (how hard is this going to burn us in the long run?) and the term length of the "loan" (will this component be around for one year or ten?).

[1] https://www.mckinsey.com/business-functions/mckinsey-digital...

> Do people in other professions have the same issues?

Yes, any domain which doesn't produce a product, conceivable to a layman in one way or another. You need conceivable results to be understood by managers and other people. For example domains producing tangible results are doing much better.

I used to be electronic engineer, and it was way simpler to explain laymen around you that you actually doing something. Blueprints, boards are conceivable more or less. Lines of code are not, especially hard to explain that less lines of code is better, since laymen don't see architecture, they see just text.

It might have something to do with (1) how expensive it is to make software and (2) the fact that software engineers are known to sometimes be purists or aesthetes or susceptible to hype.

A good comparison might be (building) architects: you know that if you let an architect run wild, there's a good chance you'll end up with something that is beautiful but wildly impractical and over budget.

(And to a lesser extent, every expert wants to do a good job, not a good enough job.)

I'm glad you've raised this, because it's something I've felt for a long time.

A few friends of mine work in accredited professions, and it's not something they have to deal with all that much, most likely because they are all accredited. They are billed by the hour, and typically if someone isn't happy with the cost the response is "okay, we'll down tools". Inversely, companies that bid on contracts will bid with a fixed cost in mind, and when estimates are taken that go over the requirements things will be fudged to fit the requirement in for that price. This means one of three things: corners will be cut, developers have to work faster, or requirements are cut. In my experience, the last point is a rarity.

I know that we software engineers like to think of ourselves as thought workers, and on the same level as the likes of lawyers and chartered accountants, but the way our industries operate are wildly different. I prefer to compare software engineering to a trade like carpentry, plumbing, etc. We share many similarities, such as easy barrier to entry and working on fixed-cost projects, but unlike trades, we're always willing to negotiate on our next pay day. You can say it's down to a difference in costs, but even on an industrial level where the person doing the sale isn't a trades-person, if a job requires £x and the client can only pay £y, then the requirements change or the job isn't taken on.

> The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

LOL, seriously? Guess you haven't worked that much with other practitioners, for example lawyers.

There is some distrust always with business relationships, and I would say that it makes sense not to trust everyone 100% all the time. In the opposite, better not to trust everyone fully, even yourself.

There is a widespread misunderstanding of what Technical Debt actually is and why it comes about. It's not about poor coding standards, it's a rational pursuit of commercial advantage through deferral of costs. I've written a few words about it a while ago here: https://datalanguage.com/features/reassessing-technical-debt

> The best approach I've seen so far is to focus on "get rid of duplicate systems"

In my experience this still requires finding the right balance, like almost anything else in programming. If you're too eager to get rid of duplication you can end up combining things that aren't actually quite the same and creating leaky abstractions that are a lot more painful than the almost-duplication was.

There's also sometimes a trade-off involved. Maintaining two very similar systems can be more work/require more engineers, but maintaining one system that is used in different ways can make you more likely to introduce bugs unexpectedly if you make a change focused on one particular use without testing all the others enough. It might depend on the particular system which cost is more acceptable.

I tend to agree, as I've seen it used and misused from both sides already. While the higher echelon often misses empathy for technical debt, what they need rather than a "euphemism treadmill" is real and actionable education on the _cost_ of technical debt.

Likewise, engineers do tend to refactor everything and their mother when left alone with too much time and with too little education on opportunity cost and context on the big picture.

The best advice I can give is first creating awareness and secondly making nonfunctional requirements part of any product negotiation. Always quantify the implicit expectation "...and it works": How long, for how many people tops, with how much possible downtime,...

More often than not, business stakeholders can get away with much less than engineers think. That way, lots of technical debt and bad examples of overengineering never come up.

But it's too late (and eventually, all code ends up as legacy code), these days I follow the school of https://martinfowler.com/bliki/OpportunisticRefactoring.html

Collect a few things that could be done way better / faster / stabler, ... — and then just convince everyone that a better architecture is the best way to get there.

> The best approach I've seen so far is to focus on "get rid of duplicate systems"

In our case (very much performance-oriented), we're actually moving toward duplicating systems, because the cost of a centralized system (in term of latency and communication overhead) has become unbearable.

My approach is to highlight technical pain points during the week and at the end of the week vote on the biggest pain point and then allocate a fixed amount of time to work to fix that in the next week.

Usually there are about 10-12, 7 of which can be safely ignored and 1-2 which are critical.

> ... because teams can use it as an excuse to work on pretty much anything!

This is a great observation that explains the uncertainty ive had from time to time.

Literally anything can be considered technical debt to someone! It's extremely vague subjective and is usually a bad reason to change/rewrite the code. Better reasons include; untestable code, scaling problems (now not predicted) and of course duplication...

> when the direct analogy is to capital underinvestment

https://www.google.co.uk/search?hl=en&gl=en&q="capital+under... (479 results)

That's because you've just made the term up. If you meant instead "the underinvestment problem", then that's actually the opposite -- that's avoidance of pursuing new features because it'll incur technical debt.

Have you managed before? Because after becoming a first-line manager, I grew a much, much greater appreciation for middle management.

> Coming up with better terms for "tech debt" seems like a good exercise

Something like "optimization" or "refining" might carry more traction.

Nobody likes paying debt, and businesses typically make payments on debt out of obligation. (Same for cleaning up.)

"Improving...flexibility" and "future execution speed" are better, but they imply nothing is gained today.

>Performing final cleanup tasks on the previous round of features.

This implies that the previous round is not actually finished

I agree, the narrative is important. Cleaning up "technical debt" may sound like a big cost and delayed eng schedule with no clear benefit for non-eng teams, despite being crucial to day to day operations.

The interesting kind of technical debt is when you realize you modeled the problem wrong, or the problem has changed and you need to change the model. A simple example is an auth system that assumes a one-to-one correspondence between account and user login. Then you get, for examples, agencies that are conceptually one account but have multiple logins (so each client can only see their own account). Oops, that breaks the model. Changing this requires database migration, code changes, UX changes, and so on. It's a conceptually simple change but has wide implications.

“It didn’t used to be this way” because of the internet. Not because the way COBOL programs were developed were inherently superior.

Once these cobol powered servers were accessible to hacking, “not changing anything” no longer worked.

https://www.computerworld.com/article/3181809/cobol-plays-ma...

In general I don't think tech debt should be communicated to the business people.

It's an internal part of engineering work, and doesn't impact users. You can think of this as "encapsulation" if you like.

Also what it sounds like to outsiders is "we're taking time off to fix things we didn't bother to do right in the first place". And that's not *100% wrong...

Back then technology could set more boundaries than today. Most of these old mainframe systems I have worked with, use batch jobs (running at night when the system is offline) today being offline is often not acceptable and the users can't wait until the next job to have their data in the system.

But I agree that being conservative with pulling in new technologies and dependencies, new and good is not the same.

I think the biggest thing with tech dept is communication. We can do this in 1 sprint but then it will cost more in the future. If business always choose the quick matter how you put it and explain when it causes problems, yeah... :(

This makes me think part of the problem with tech debt is that it means different things to different people. Some people might describe it as a codebase just not keeping up with this month's latest and greatest JS frameworks, but I think of it as something much more nefarious- low level shittiness and incoherency that tends to seep into long-lived large projects over time. Eventually the lack of refactoring to account for new features or behaviors turns it into a giant blob of spaghetti, where everything knows about everything and what should be minor, localized changes can have unforeseen consequences. This is harder to deal with IMO because it's something that appears so slowly that it's hard to stop in isolated instances ("WTF, you wanna rewrite this whole class instead of just commenting out that one line and adding one more tiny code path to fix this bug right now?!?"). That type of technical debt is more of a boiling frog problem where it's not obvious to everyone how bad it is until serious consequences have started to arise.

Java is pretty good at this. Before jigsaw we had projects that we hadn't touched for 13 years and they still compiled on the newest versions.

There's positives and negatives to this of course.

> COBOL on a mainframe would remain current for decades

Only if the underlying problem solved by the program doesn't change for decades, and even if this case, the piece of software in question is a constrain to the industry using it. I'm sure a dentist would loves to have some fancy REST 4.0 BUZZ-WORD-OF-THE-DAY client management system.

I am using some industry "standard" industrial CNC software and it not only constrains me to 15 years old hardware but also restrict me to use programming quirk specific to this software which very much qualify as being "tech debt". In this case, the business is built around the software.

There's really a few distinct concepts here, and I think it's important to be precise.

- Keeping up with platform churn. How much of this you will do is really a CTO level choice, in terms of a) permitting internal platform teams to break backwards compatibility, and b) permitting adoption of trendy technology from outside.

- Knowingly trading off quality for speed to meet a deadline. This is tech debt in the strictest sense and you can quantify it as you accumulate it. Tech debt comes from a business / product management choice to force a project to release before it meets standards.

- Seeing better abstractions and decompositions, new test cases, observability hooks, etc. with the benefit of hindsight and experience. Examples: we found four bugs in this module this quarter and it's really complex, I see a simpler and less error-prone way to rewrite it. There's a lot of interest in adding new features with a certain shape and each time you have to touch code in three different places, let me consolidate those so you only have to touch one. People keep asking a particular question that requires painstaking correlation of different logs, let's add a new logging topic that answers it directly. Stuff like this. These are never strictly necessary and business impact is hard to quantify, but if you do this stuff never, you're going to have a lot of cruft in a few years.

I’ve recently been reading ‘A deepness in the sky’ and it has the concept of ‘Programmer-Archeologists’ exactly for this reason.

It’s set many thousands of years in the future and I have trouble imagining how many old versions of packages and libraries would exist at that point.

I've got Javascript projects of a couple of years old. There are certainly things I'd do differently now if I were to set them up today, but I wouldn't consider that technical debt: those things are not giving me practical problems in the foreseeable future.

The only of such projects that I'm still running which does have significant technical debt, from my point of view, is the one I set up without a CI system and without a package manager with a lock file (those didn't exist back then) - I cannot update that code today, because I cannot build it. Rewriting that one with a reproducible build set up is something I consider tech debt and which is still on my TODO list of intending to solve one day.

You shouldn't prioritize buying a fire suppression system over putting out the fire that is melting production :)

Those consequences can be small and extremely delayed if interest rates are low.

That's how I think about it, and that's hopefully how a lot of people think about it, but I suspect for most people that's not true. For a lot of people debt isn't nearly as scary as it ought to be, and the idea of, instead of paying off debt, spending money to theoretically make more money seems like much more of a sure thing than it actually is.

Perfect example: I'm in SV, and I bet there's a not insignificant slice of the population around me who think that the cure to budget deficits (i.e., debt) is to reduce income by cutting taxes, based on a mystical belief never seen in reality that the growth fueled by the tax cuts will result in both lower taxes for the population and higher tax revenues for the government (i.e., investment).

I'm almost 50 years old, and there has never been one iota of truth to this notion in my lifetime. Not one teeny tiny iota. But some people just can't resist going back to that well over and over again with the same results every time. The notion that you can kill two birds with one stone, and both ignore debt and create wealth at the same time via magic Austrian fairy dust must be very powerful. It's like a black hole of self-evident idiocy that people keep flying into for some reason. It must be a form of cognitive bias, but if so I don't know what to call it.

Well, paying off debt can often be deferred indefinitely too, as long as you keep paying the interest.

Instead of thinking of it as "debt payments" you could just think of it as high operating expenses for the development team, and invest in reducing them.

I sort of agree with the author, for a similar reason that I don't like the concept of code rotting. Digital artifacts left alone don't change. They don't become more broken over time. Technical debt in code you're not currently modifying isn't incurring interest - and a debt without interest is one that you can safely leave unpaid indefinetly.

Both "code rot" and "technical debt" are concepts that, unlike their real-world analogs, are only problematic in context of ever changing environment. I feel this is a crucial and unmentioned difference that makes the analogy bad.

It should be quantifiable to within an order of magnitude or so, although possibly not in advance. In my experience the problem is that any individual piece of technical debt is too small for it to seem to be worth the effort for the difference knowing the value would make.

Occasionally the new feature's debt is never paid. I'm cleaning up a location which has had this happen 30+ times over 20 years. It resulted in 15+ sightly different ways of doing the same thing. Much of this without a function call in sight. I've been splitting functions out of it slowly reducing the 80+ variables into discrete behaviors. It's now unit testable.

I'm often asked why can't I add something quickly, pointing to this file is more than enough reason for all but the most important projects to agree to wait.

What should I call this if not Technical Debt?

Or the company can't source the same materials / quality as before (supplier change, company moves, etc) and need to go into trouble fulfilling existing orders with the old design (paying extra both time and money for the old material) while trying to sort out a new design with the new material available...

How did you approach your role as PM, aware of the tech debt but also reporting to an unaware CEO? Power through with new features at half the speed, or try to convince the CEO to allow refactoring/maintenance?

Eating your own dog food is a reference to https://en.wikipedia.org/wiki/Alpo_(pet_food)#Marketing I’m surprised marketing guys don’t know that

That version of Scrum is horrible, and should be fixed. Scrum switched "commitment" to "forecast" in 2011, and it should be well known by now that you can't compare velocity across teams, or even within the same team out past a couple of sprints.

Or dependencies.

Yes, it's extremely strange that people are looking for new terminology because they've been using the existing, perfectly good, terminology incorrectly.

How's that meaningfully different than performance and code cleanup cases? Frequently, a team know how to implement a feature in an efficient manner, but only has time for a low-performance hack job. Similarly, "code cleanup" is as much a code word for "not how I would have done it" as it is for "bad architecture that will start constraining us very soon".

I’m dubious of defining technical debt merely as the result of previous decisions taken to trade off delivery time for maintainability. That’s the responsible form of technical debt and responsible people will incur it. But there are irresponsible forms of technical debt just as there are irresponsible forms of financial debt. What makes something technical debt is the “interest” in terms of maintainability that one pays for it.

I think that may companies need a fundamental shift in managerial culture before they take technical debt even remotely seriously.

The only way I've ever managed to do anything about technical debt was by hiding the fact that I was doing it - since it was completely impossible to get any level of managerial buy in or even permission.

It shouldn't have had to be that way, I should've had higher level support, other developers to work with and competent testers - working in that situation ultimately left me never wanting to write software for anyone else ever again.

It's a disease of reckless incompetence, and it permeates the managerial structures of large swathes of the software development industry.

To make the term more useful to me I like to extend the analogy and think in terms of interest paid on technical debt.

So you took a shortcut because of pressure to ship but now you’re spending loads more time providing tech support. That’s the interest you’re paying on the technical debt.

So in terms of choosing what technical debt to pay off first consider which ones have the highest interest payments. And remember that like financial debt, technical debt also sometimes comes with introductory interest free periods.

> Always pay down technical debt when you can. It will be paid eventually, it's just a question of how painful it will be when you do.

Not really

I have a program which is a right pain, and takes an hour to do a yearly task. To clean it up would take about 200 hours.

The program is unlikely to be around in 20 years time, let alone 200.

Your third example is just some combination of the first two. Performance issues: the efficient solution is too complex, we don't have time. Code cleanup: this is not the way I would've done it if I had more time. The debt metaphor fits so well because you're borrowing time.

Debt can also be taken on accidentally. You can build a couple of features one way and expect future features to be similar, but then reality bites and the way you've built them makes the next feature that happens to come along harder, and the next, and the next. Or the way you've built them doesn't allow for the rate of change in that area - something in code wants to be in config, or something in config wants to be in a CMS. In the debt analogy, it's like having a variable interest rate: you can make a decision which is correct for today's conditions, but the market moves and you're paying twice the interest and suddenly it doesn't make sense any more.