Hacker News Reader

Comment by marktangotango | Hacker News Reader

After the MVP is found, then the real work begins in building a company. We all understand that if everything is a priority, then nothing is, but often times, business resources don't. Support wants bugs fixed to keep customers happy and to re-sign. Sales whats features to land their whales. Engineering wants to work off tech debt, refine, and innovate. Etc.. I've found that a formal "Intake" process works wonders. Engineering needs an advocate to take these disparate priorities and refine what, when and how to address. It's really that simple, and that complex. Finding someone to own it. Sometimes it's impossible depending on the Org and personalities involved.

View on HN

> Engineering wants to work off tech debt, refine, and innovate.

I know I'm not the only one who is backing away slowly from the concept of tech debt, and I'm not the only one trying to drag other people with me.

Tech debt has turned into hippie-dippie bullshit in the minds of many people. It was an attempt at enlightened self interest by appealing to the bean counters, but if you're talking to bean counters you've already lost.

The developers want easy things to be easy so they don't have to spend half their time slogging through code that does everything the wrong way, half their time explaining why everything takes so long (heading toward infinity), and half their time implementing new functionality. And yes that's 3/2 which is why we have people doing 60 hour work weeks.

A long time ago I saw a speaker claim that it's not tech debt, it's wear and tear. I don't know if I agree entirely with that characterization, but I definitely think that it's less wrong.

Tech debt is too general a term to be useful.

A lot of what people call tech debt is just working code that they don’t understand (maybe because all of the people that did understand it are now gone). Or it’s complex, which may just be because it’s dealing with a complex issue and accumulated lots of bug fixes over time. Or tech debt is just nitpicky design choices which may very well be not-great but aren’t actually causing any problems. None of that is worth investing in beyond a fix-as-you-go, in my opinion, due to there usually being problems to solve with much greater business impact.

On the other hand, you have tech debt like: a complex set of dependencies/services with poor interfaces that makes what should be simple changes very complex and risky. Something implemented so inefficiently that it’s slow enough to impact UX or wastes a lot of computing resources ($$$). Or code that is just so, so bad that making any change is like walking in a minefield - and while a lot of engineers are way too quick to put code in this bucket, I do think it’s possible (eg a complete ball of mud application with tons of global state, very poorly applied concurrency patterns, extreme lack of parameterization in favor of copy-paste).

That said, I have often seen “tech debt” used as a rationalization to work on some nitpick with no business impact and without a clear improvement. In my opinion, this is partially because working on such tech debt can be very easy: it doesn’t require understanding more complex business requirements, it might be very easy to test with existing integration tests, it has little risk to production, and it doesn’t have task/planning dependencies with other people.

Here's the stance I take: there are many reasons why something might be labeled tech debt. However; assuming the remediation of that debt is implemented in good faith, by reasonable skilled engineers; there is near-always business value in remediating it. It may not always be the same $/hr return on investment, but there is usually positive business value.

Secondary point: there is "near-always" positive business value; near-always; and its actually impossible to predict if a given tech debt mitigation project will result in negative business value. You can only know in retrospect.

An example you give and label as bad: some crusty old system, outdated language, legacy, everyone who knows it is gone. But, it works.

Mitigation: rewrite it in something more modern that everyone on the team knows. Do that, and out of thin air your organization just minted a handful of new Experts on the system. They're forced to dive into the old code. Document the new stuff. Projects like this can be pretty fun!

Counter-point: The old system works. Why change it? We just need engineers to dive in, learn it, keep the old crusty libraries and frameworks up to date, lets hire and plan for that. And people fuckin wonder why engineering retention is so bad! "No one who works here knows how it works" Huh I wonder why they left! Let's pass all our problems onto the new generation; they don't get to own anything, their job is to fix shit and bolt on new functionality at a tenth the rate they feel should be possible, enjoy being an indefinite renter of the system until you get as fed up as the generation prior, move companies, and do it again.

Business value? You're providing cool projects to your employees. That's it. We'll make up reasons it benefits customers. "We'll be in a better position to scale and reduce API latencies" bs. "Feature velocity will be higher" maybe. Throw out some more tactile reasons like it'll ease hiring because when people hear PHP they run for the hills. Yeah that's more correct, but the biggest reason is expert retention. Have you got experts? If yes: don't rewrite. If no: maybe rewrite. Let engineers rewrite, they feel ownership and responsibility over the system, they probably stay longer. That's it.

"Complexity" is similar. Its complex because the business problem is complex. Sure! Engineers just need to dive in and learn it. When? Is the organization willing to literally block off time for the engineers to do this? Lol. Has anyone ever worked at an org that did this? It could work! But what they may block off time for is: rewriting it to make it less complex. The end result probably won't be less complex. But you'll have gained an SME! And the organization can feel good perceiving the problem area as "now less complex" (whereas in reality all you did was train up an SME via way of a refactor).

When does it go wrong? Obviously; if a legacy system works, or solves a complex problem, and the rewrite doesn't quite capture what it does. The old code is right there; that's the spec. If the new doesn't match the old spec, it probably comes down to someone not fully understanding the old code. Guess what: that was gonna cause a problem in the future. You just expedited and uncovered a potential problem; but in a context where it can usually be remediated significantly quicker because you're building off a new system with tons of experts, rather than an old system no one likes working in. That's business value!

And sure; there are much rarer cases where the entire project goes way over time, the new system sucks, gets caught in development hell, people leave mid-project... that sucks. It can happen. But I'll reiterate: its literally impossible to know with any reasonable certainty that this will happen before taking it on. This means its an extremely poor reason to not to do it.

To be clear: I am not asserting that addressing tech debt should always be a priority. I'm asserting that there is near-always positive business value, and that People are extremely bad at estimating what that positive business value will be because it doesn't have a dollar-sign attached to it like CS and Sales orgs have.

> just working code that they don’t understand (maybe because all of the people that did understand it are now gone).

Well in that case the real debt is the lack of documentation or clarity in the code. If the code is not written in a way that someone other than the original author can easily understand it, that's a problem. It also indicates gaps in the process of transferring ownership. There should be a team of owners for each piece of code that brings new members up to speed when people leave. Knowledge mustn't leave the company when people leave.

OP here. We had a lot of discussion about where Tech Debt falls on the spectrum of projects to work on. That merits its own post. But the gist of it is that resolving tech debt has value. The challenge is that most engineering organizations are awful at describing and communicating that value.

For example: "If we refactor this code and migrate to a new server, we'll be able to deploy in 15 minutes instead of 6 hours, and we'll be able to ship features faster to customers with lower risk. This means that we'll get back ~90h of engineering time per week (6 hours x 5 people x 3 teams), and our defect rate will go down by 50% (We see 10% of customers churn due to preventable defects).. Therefore, there is $702,000 savings in engineering efficiency (more time available) and would decrease churn by 10%, resulting in [X] more value retained."

Once our engineering team started thinking about the business case for why tech debt should be prioritized, they started to make really compelling arguments for why something needed to be worked on, and we could evaluate it against explicit business/feature requests. This value rolled up into the form of decreasing costs or increasing revenue.

Again: resolving tech debt has value, and the engineering leadership needs to work on figuring out what that value is and how it stacks up against all of the other valuable work that the company wants to do to grow. Engineering leaders who don't understand this often struggle at the executive level and ultimately fail the teams they represent.

The problem I think is engineering haven't figured out good ways to measure software productivity, and so it's very hard to make the kinds of estimates that we need to properly justify work that otherwise delivers zero immediate business value.

(I also suspect that not all refactoring work is equal, and that most refactoring work don't result in any positive engineering or business improvements.)

This is very fair! We used a few guide posts:

- 1: Keep asking "why is this valuable" until you get to the money questions. It's often a few layers deep. Not being able to get at that (missing an ability to quantify) was in itself something that engineering leadership needed to work on. After all -- how can you prioritize between two things if you can't figure out what value they create?

- 2: Most engineering teams know that they might be able to save some time by refactoring. We figured out the average hourly cost of engineering effort and used that as a multiplier for time saved by a new project's implementation. You can take it a step further and subtract the cost of working on a project from the time it ultimately saves you to remove "dumb" refactors that take longer to deliver than time they save.

- 3: If you can't get to a tech debt's value, that's a red flag as to if it actually creates value. This was true for features and business requests as well. We also said no to features that "felt good" but ultimately created zero value (some redesigns included). "Rewriting something in a new language" because a new sr manager with strong preferences joined the team was a big culprit of these sorts of projects.

- 4: We also often asked if a tech debt project needed to be done independently of other work, or if it could be included as part of a new feature that we were working on. An example was that one team pitched rewriting a microservice because it had no API. On the other side of the company, we had a project to get rid of that feature entirely and replace it with a different functionality. If tech debt projects were handled entirely within the domain of engineering, we would have spent weeks rewriting something only to throw it away a month later. Hence the need for bubbling tech debt up to the single stack rank, and thus treating tech debt projects the same as any other project we were working on (with appropriate visibility, buy-in, and evaluation of value/urgency).

> "why is this valuable" until you get to the money questions

Often this is one layer too far though. Business relationships aren't 100% about business, or we wouldn't put an adjective in front of 'relationship'. We'd just call it 'business'. If you know engineers are trying to put together a proposal to take on a project that might not be immediately obvious to the business, we already know all we need to know here. Developers don't go out of their way to put together proposals for frippery, and most of them don't put together proposals for clear business cases either. They put together proposals because something is pissing them off, one way or another.

If you say no, after they've stuck their necks out, then you've damaged a relationship with people who are not comfortable being clear or direct about these sorts of problems. People who you are paying to more clever than you. Which means when the consequences come, you won't see them coming, be it by apathy or malice.

We aren't allowed to say, "if you don't give us time to fix some of this stuff I'm going to quit, or worse," and so everyone tries to sugar coat the real reasons with things that are objectively true but specious motivations at best.

As far as I'm concerned, the original notion of tech debt was intended to be about borrowing on good will, asking people to do something they don't really want to do, but with the idea of an eventual end. That kind of sacrifice can often be quite healthy, even transformative. But what happened instead was advanced game theory. How long can we stay on the verge of social and/or technical bankruptcy without ever crossing over? And every new kind of 'check' they learn to float is a whole new world of delaying tactics.

Getting a divorce is very expensive, but you don't avoid divorce because it's expensive. If you're smart, you don't even mention that it's expensive, because if you weren't having an argument before, you are certainly having one now.

Reading these comments and replies has me very interested. I think talking about technical debts and ways of quantifying it is definitely worth another post. I'd read it!

You might find "The principles of product development flow" interesting. A large component of the book is about quantifying all work to be done, and then managing the resulting queues and WIP. Many aspects of the article reminded me of this book, but with concrete implementations of general techniques, which was very helpful.

[1] https://www.thriftbooks.com/w/the-principles-of-product-deve...

> I also suspect that not all refactoring work is equal, and that most refactoring work don't result in any positive engineering or business improvements.

We have lots of old code in production, some 25 years old. Much of it written during the startup crunch, with little thought to extensibility and so on.

But it works.

So we have a rule to only cleanup/refactor code when we have a specific case which requires us to change it. If it requires a bit more than a minor change, we'll do a cleanup, refactor or rewrite depending on the circumstance.

Remove-tech-debt-as-you-go is how I approach the problem. Doing so ensures that removing tech debt is not done in a vacuum, disconnected from business needs.

Easy, you just measure the lines of code added. More code means more productive.

The one checking in the node_modules folder is the most productive developer?

My software work is centered around security and I find it almost impossible to persuade managers that security is worth anything.

The exception is when a major hacking event unfolds in the media. Then, suddenly everyone is attentive.

I suspect it is the same with militaries, except the USA and a few other countries. It is easy for politicians to cut military budgets and spend the money on civilians, and some intellectuals will even praise them as prioritizing peace, until suddenly tanks adorned with white letters roll over Ukrainian borders. Then, countries panic because their underfinanced military force is inadequate to the threat, which just became very real. (I am looking at you, Germany.)

Once the engineering team started thinking about business cases, they started wasting time they could be dedicating to engineering. I know where you come from, on a similar journey, but when things don’t work, adding things to do is the worst option.

I'm not sure I agree. There's two key elements:

1. If you don't properly evaluate the work and be selective with what you work on, actual engineering time will be allocated to too many projects. People spin their wheels working on potentially insignificant things, which is a morale killer. So thinking about the business cases was a way of reducing work, not adding work.

2. I tried to emphasize that thinking about business cases is something product and engineering management/leadership need to own, not necessarily IC engineers. ICs can add ideas to the backlog and identify opportunity, and it is the job of PMs and EMs to evaluate those ideas and sequence them for being worked on, relative to other projects. The feedback loop that is important to ICs is the impact of their work in terms of user metrics and goals the project was supposed to accomplish. I don't want to get too heartfelt about it (because Capitalism, Yay!), but there's a difference between churning out features into a black hole and knowing how your deliverables affect the people for whom you build things, and letting that continue to inform your work.

I think we are more aligned than what our exchange may superficially imply. But to clarify, from a value generation perspective, I don’t see tech debt as something whose payoff would generate any value other than in terms of technical flexibility and long term engineering resources availability. If a tech debt item is adding a feature then it’s not tech debt. Tech debt is born out of suboptimal support for features. So the feature is in place, but not in the proper technical way (I.e. against target architecture, held together with manual processes etc.). As such my approach with tech debt is realise the value by having cadence of paying tech debt off, not being afraid or thinking too much about the value of bringing things up to date. It’s more of a keep the engine running approach, at least in big engineering organisations where that makes sense from financial perspective as they resemble more a locomotive than a car in economics of restarting vs running ongoing. I am also confident both approaches may equally work or fail depending on organisation dynamics.

My analysis is that the problem is "tech debt is bad". Just like real debt, there are times when you sign your name to it on purpose because it's a good decision. For example, early on in a project when the requirements are fuzzy and undiscovered.

Like financial debt, you pay interest on tech debt, so you do want to strategically pay it down when the stars align, maybe you've recently put out some new features, and you can buy the political capital with customers (and by extension, management).

Debt isn't a bad analogy but an appreciation for the nuance would be nice, I don't commonly see that.

Technical debt is unavoidable, but letting technical debt grow without managing it is a great way to kill a project.

Explaining the benefits of paying down particlar pieces of technical debt is important, but another important piece of the puzzle is talking in advance with stakeholders about when, where and how much technical debt you want to take on to launch a product. When the existence of technical debt isn't talked about until it needs to be repaid and thus comes as a surprise, the conversation becomes a lot harder. Additionally many of the decisions about how much technical debt to take on to launch something can be improved by bringing stakeholders directly into the conversation, at least on a general level.

There’s a whiff of victim blaming here that bothers me.

When taking on tech debt you often get a lot of pillow talk. We have great reasons to do the wrong thing for the right reasons, but that is all forgotten after the crisis is over. Either immediately for those really bad at it, or when the next crisis hits.

I do agree though that when neither transparency nor awareness are good, there will be several crises in the middle that are triggered by the tech debt. I’ve seen that play out a couple of times, typically when I’ve joined a project in progress, and still building trust before I can start pointing out brutal facts.

Some developers are blind to different forms of tech debt. And as long as there is dissenting opinion the business will doubt any information that suggests they can’t have their cake and eat it too. I’ve known a handful of jaded engineers at least who, instead of being afraid or awed by titles, end up treating managers like willful children. It… does not yield consistently good results, but then again I don’t know anyone or any process that does. I’ve seen worse, and I’ve seen better.

> There’s a whiff of victim blaming here that bothers me.

Huh, what are you even talking about? What victim am I blaming for what?

I am sharing an important part of the technical debt management process that I don't see mentioned as frequently. Planning out where and when your organization should take on technical debt seems as important to me as planning out where and when your organization should pay down that debt.

Treating people like children is rarely a good approach and to me indicates someone who is either lazy or unwilling to take the effort to communicate well (edit: including when working with actual children).

If an organization can't resolve dissent effectively, that seems like a deeper underlying organizational disfunction that will cause all kinds of problems.

The truth is, the line for what is technical debt is kinda fuzzy and a there is a lot of other stuff that isn't really technical debt, but rather process or organizational debt.

The problem with "tech debt" is that in the heads of non-developers it is equated with financial debt. When you take on financial debt for a business it's a long-term investment (a buy-to-rent mortgage on a property, or a business loan to buy a factory plant for example). So OK, we're a startup that needs to find the right market, so we'll take on X amount of tech debt to get there, and then when we are in a better place we'll pay down the tech debt in installments of X amount per month.

Mostly what I've seen of "tech debt" has not been this kind of deliberate decision making of tradeoffs. It's more like:

1. Non-tech founders want to start a tech company.

2. Seeing the high salaries of developers they opt to hire their college roommate or relative.

3. Friend/relative is horribly inexperienced and builds a pile of crap.

4. Said pile of crap nevertheless gets market traction.

5. Customers complain of endless bugs & sales want long list of features.

6. Friend/relative is promoted to CTO. Time to hire experienced devs!

7. Experienced devs say "this is a pile of crap, throw it away and start over".

8. Founders tell devs no, you have to fix bugs and add features. Hop to it.

9. Developers have no leeway or political capital to fix the underlying problems, do what they can, and eventually leave.

So it's more akin to an 18 year old with a gambling habit who has just discovered payday loans.

At some point you have to decide if integrity means knowing the right thing to do and doing it, or knowing the right thing to do and asking permission to do it.

If you can’t get your coworkers on board, then they can make you look bad faster than the project will reach an intervention. If you can, you can stop having abstract conversations with managers about being Good and just start telling them that this is how long things take now.

Relentless refactoring is a set of skills that are interesting but difficult enough that people lose the plot. It’s really about amortizing all difficult work across all of the work, not just the difficult bits. This is actually a strategy from real-time systems, where maximum reaction time matters, and something that XP knew and Scrum does not. A thing I’ve noticed is that there’s often a lot of gaming by management once the engineers start to say “no” but haven’t mastered yet. The bargaining is expensive in time, energy, and good will (once people realize they’re trying to trick us into doing something we said was hard by going at it sideways).

When you add a day, two days, or week to all estimates, then the ratio between the smallest and largest stories decreases. The incentive to waste time trying to trick people into saying yes to something that violates the project triangle is drastically reduced. They won’t like it, but they weren’t liking anything else we tried to tell them so that’s not really a new situation. What’s new is that they see we’re serious and they don’t like it.

The counter-point here is that while you can argue about the name, there is a very real issue that poor architecture will slow progress not because it's "hard" to implement features, but because organizationally it becomes impossible to get them merged.

We have an endless parade of commentary on HN about what and how much should be in commits and pull requests and what have you, but the thing I've always found credulous about these ideas is how they survive contact with a code base where a "small" feature is going to have a very wide-ranging pattern of changes necessary to support it.

It doesn't take much before their are features you functionally can't implement, because there's no possible way to build a neat set of commits for them and get that through reviewers - particularly if there's a lot of activity, because the immediate effect is usually that you'll break a lot of other minor patches with the changes (or they keep breaking you).

I'm not sure though that I've seen anyone actually prioritize fixing technical debt in these terms though.

I am using a low code/no code tool for some back office services and we're investing into the enterprise version. The enterprise version is on-prem only. To migrate from cloud to on-prem, we have to export the cloud data to the on-prem service. Each "section" has a download to json button but there is no download all button. So we have to go to all 50+ sections, click download, upload to our on-prem server, and click in a few places to "reset" things.

The fact that these guys cannot build a migration button smells like major technical debt. These guys aren't stupid since they have built an incredible tool. Somewhere along the line, their product became fragmented enough, where it would be too costly to build a feature that would lower the barrier to entry for their enterprise service. We're talking about high 5 figure to a low 6 figure yearly contract. They're telling us we need to go click for an hour to do the migration.

So if you don't think this is technical debt, I'm curious as to your interpretation of the situation.

That seems like self sabotage to me. Unless their on-prem solution is less expensive, wouldn’t they make more money if the process were simpler?

Some software tasks are tedious and developers try to avoid doing them. But externalizing them to your customers is not a great strategy.

Excellent point. If you have no other option than to do this manually, it sounds like a great use case to automate this problem with playwright.dev.

I've found myself holding a bit of chagrin at the term 'technical debt' lately as well, mostly because it feels like saying it, pointing to it, quantifying it, and shoving 'it' under people's noses and proving its there (not to mention the usual ceremonies of what I'd like to do about it) isn't doing me any favors anymore.

And at that point....

....I don't know that simply calling it something else is going to help.

Yes I'm a bit jaded, grab a drink, next one's on me

Begin by defining technical debt. Write down an actual definition for the phrase. Once you have a working definition that everyone agrees on you will find all your problems with the term will disappear. (Hint: good luck finding an agreed on definition).

Tech debt and refactoring are the two most overloaded terms in software development. Most often "tech debit" isn't - it is code that met the need when it was written, but the need changed later. Wear and tear seems much more appropriate.

And re-writing from scratch isn't refactoring, but that seems to be the most frequent usage of term these days.

> A long time ago I saw a speaker claim that it's not tech debt, it's wear and tear.

It sounds like you agree with the idea in general, but just don't like the current metaphor. Can you expand on what about this alternate metaphor fits the phenomenon better?

I can. Even if you design a good system up front, the requirements graph changes over time until your great ideas have turned into hindrances. This can be scaling issues, but mostly it's where reality turned out to be different than your abstractions were built for.

Maybe you made one module very generic but now it's cumbersome to add features to. Maybe you didn't add enough flexibility in another place which resulted in a buildup of suboptimal code. Maybe extreme concurrency was planned for but never needed and now the person has left and no one else understands how that part of the system works.

So now, over time, this emergent phenomenon that some call code rot causes your beautiful, elegant greenfield project to turn into a monster that no one wants to touch. It's hard to modify, easy to break, and takes forever to work on. It's legacy code.

I've begun to see tech debt as business processes that are too expensive to replace. You can either encapsulate the process with modern thinking or sell your company to someone who has investors willing to pay to create new processes.

Just swapping tooling shouldn't be considered tech debt if you can actually do it without breaking income.

What you've just described is management.

There should be someone in the company, with visibility across the entire organization who says "yes" and "no" and explains why.

I've worked in places like this and the clarity is amazing and very motivating. The goals from the top down were 3 things. Everyone in the org could understand and apply it to their own work. Anything else was deprioritized. There was no confusion or swirl.

Lots of people piss on management as useless, but when done well, it can contribute a lot in terms of work-life balance, employees feeling motivated and organizational cohesion.

Done poorly it can actually make things worse.

Think you nailed it by breaking it up by groups of people with different lists.

Literally every group/dept/team has a list. Someone/process has to arrange that into a single list for the group that builds. If you have more than one group that builds stuff, then more output lists and more complex the transformation from input lists to output lists.

Guess this is a simplistic summary of what PMs and Eng managers do.

This is one of the main things a good leader should be doing. Not just execs, but up and down the chain, one of the most valuable things a leader can do is get everyone pointed in the right direction. Make decisions about priorities, communicate those decisions to the team and the rest of the org, and then commit to getting them done. Ruthlessly prioritize and be willing to say no.

It's not actually easy, for a number of reasons. One is organizational. If your boss isn't doing the same then then it's going to be harder for you to do it. If you're responsible to multiple people who don't have any common chain of command then it's going to be really hard. And it involves saying "no" and standing your ground.

Reply on news.ycombinator.com