Using Technical Debt in Your Favor
levelup.gitconnected.com
levelup.gitconnected.com
I think this is both an art and a science, and the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
There are so many factors involved that it's not surprising so many projects accrue horrible debt.
I couldn't have stated it better myself. Most of the time it start from culture, leadership and then experience etc.
I have worked on multiple projects with varied amount of technical debt. But the project with the most amount of debt happened because of poor leadership.
The guy who was managing the project was more interested in currying favor. The end result - everyone's salary was being shared across the company in un-encrypted plain text format.
It reliably fascinates me that in spite of more than a century of history, management culture lacks both words and concepts for many common and predictable organisational failure modes.
Does it? The manager in GP, for instance, seems to be a textbook example of a thermocline of truth.
> the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
>the project with the most amount of debt happened because of poor leadership.
If your system started out with tens of requests per second, and it was designed for that and it works fine under that load, having it break if the load increases 10x is not technical debt
Or it might not even have existed (or was unstable) when you started work, but it got better later.
I'd also urge you to use higher-level languages from the get-go. If you can use Rust instead of C, do it. If Haskell is appropriate, use it. These are all choices that are fantastically easier to make early on than later.
Its the same readon why I almost always immediately use vue or react. Because it doesnt cost me much (~0) but if that thing has to scale i’ll be ready
Obviously there's a bit of YAGNI in tension here.
But if you have the experience to write it correctly from the get-go at low additional cost, then it's worth doing that instead of risking a massive rewrite later.
I have never had any problem with putting some code into a library after the software is done. It is almost always a 1-4 hours exercise, and when it's not, it's a 2 days task at most.
Async code is also something that I have never seen it being a huge problem. Mostly because only a small part of your code needs to be async, most of it can stay synchronous, scheduled by the small async part.
Thread safety is a real problem. One can indeed trade some of it for some development speed, and this can become a huge bottleneck later/soon.
I used to think this was true, but after talking to architecture/civil engineering friends a lot about this, I've learned that this is less true than I believed. There are often changing requirements for buildings due to different use cases, changing regulations or building codes, or a change in energy prices (or the social cost of using a lot of energy). Sometimes new technologies are used without a lot of experience with them, leading to high operating expenses for the lifetime of the building.
I asked my friend about your comment and he gave me a few interesting examples: 1) "flying form construction" that leads to exposed concrete slabs on buildings that bleed heat in the winter. This was really cheap to build but turned out to be really awful for energy efficiency (once we started caring). 2) a building in the 50s-60s that was built with automatic window blinds. The maintenance cost of fixing the blinds when they inevitably broke led to them not getting fixed, and the building becoming basically useless.
If you're interested in reading more about this, Stewart Brand's "How Buildings Learn" is a really good read for laypeople. I was really surprised by how much of the book could be applied to software engineering.
(This is not a field I'm familiar with so any errors are mine, not my friend's.)
They have now finally decided to move the MPs out so they can do the refactoring they've needed to do for years...
Sometimes, supporting the legacy system can be technical debt. But sometimes when people try to replace the legacy system, they spend a lot of time and money and end up costing way more than simply maintaining the legacy system. This is part of why some nuclear plants will continue to operate on PDP-11's until the year 2050.
Technical debt isn't really debt if it's actually just a part of the cost of doing business. You design your system for its intended purpose, and if it's accomplishing that purpose well, you don't need a redesign. Don't break what doesn't need fixing.
When that's the level of awareness the average development team is dealing with, there's limited practical purpose in exploring the nuances of healthy debt management. It's more useful to talk about technical debt like nutritionists talk about empty calories and transfats.
The same _can be_ true with code. There can be a better way to do something but if you cannot afford the time to develop and iterate this idea then it can be advantageous to develop the simplest solution that meets your needs today and make progress.
Regardless of how much PMs might want things or fiddle with the schedule, as professionals it's our job- not theirs- to say when things can't be done, shouldn't be attempted, or might cost more than they're worth.
That gets back to company culture (which others have identified higher up). Not every organisation responds appropriately when developers push back (either due to imposed deadlines or office politics).
In my experience, organizations generally do the right thing if all the developers push back. The problem is that you can always find developers that will push a hack or deploy an overwrought solution for a simple problem.
Maybe you're lucky or maybe I'm benighted, but it's rare that an organization consciously decides that this is the quarter to finish off the Java 5 migrations, even if that means delaying the Jabberwocky Project and even if there's a non-zero chance of customer impact.
Partly because it's a really hard problem to attach an accurate and objective number to the value of something like that.
> ...technical debt and the benefits and costs associated with varied approaches...
That's not entirely accurate. Engineers are experts at technical benefits of different approaches. And they should be experts on the entire cost of the systems they are responsible for. But they're not necessarily experts on the benefits. They aren't necessarily qualified to anticipate what different features and different release schedules will do to the revenue for the product. In B2C companies with millions of users, you can run statistics and A/B testing to make this stuff more objective, but a lot of companies live and die by big sales to a smaller number of customers. So determining the benefits of shipping a month early depends a lot on knowing particular people well enough to guess how they'll react to good or bad news.
One solution I thought of was to enact a 20% code debt policy, similar to google's 20% time but for code debt. Hey developer, is there something causing you pain? Fix it!
That's sort of how I do it. The key is to never tell management explicitly. It's just normal part of the job.
The only problem is that this setup is subject to the tragedy of the commons. If it takes 25% more work to make a change cleanly, the developers throwing hacks around look 20% more productive if we ignore technical debt. And, worse, they're often the only people in the world that understand the hacks, so they get to be the hero-experts that come in and save the day when everyone else is trying to figure out what the hell is going on.
So even if we'll do our best to bake the cost of cleanup into our day-to-day tasks, that only works as long as a critical mass of the developers in the organization do the same thing. Otherwise, you'll have a full time job plus some just cleaning up messes.
This works only if you own your part of the code for a some time so your investment can pay off.
There are two outcomes (1) they trust you, and grudgingly accept your estimates. You address what debt you can and ensure padding doesn't increase the next time thus trust is maintained (2) they force a timeline on you, in which case when it's late or broken in production you can at least say you needed more time.
There is a third more egregious possibility, that they get an alternative estimate from elsewhere or one of your colleagues and in my experience this is the principle weakness to standing your ground "we'll take our business elsewhere" and I am truly sorry for your trouble if that is the case. Hopefully you have secure employment and the whole mess then just becomes the problem of the person that undercut you.
Of course, in all these situations you have to make a call based on your personal/organisational circumstances but I just want to say that ultimately being upfront, diligent and continuously demonstrating trustworthiness may not always pay off in each particular instance, but over the longer term as reputation accrues they do.
Can you give an example?
Something unexpected pops up like a family member dies and you need to arrange a funeral, but don't have the cash on hand because some other emergency last month used up your liquid savings. You know that you can pay it off over the long run, but don't have any other way to deal with it right now. So you go into a small amount of debt that you can likely pay off without paying too much over the original cost.
There are a million ways that they can be good. There are probably more ways they can be bad, but this question is very simplistic.
Why bad for you thing is actually good for you. or Why thing everybody likes is not so great after all.
Is everyone on hacker news just a compulsive contrarian?
No we're not.
Whenever I'm told that I'm not allowed to disagree for some reason or another, I immediately stop and reevaluate my views. Usually they don't change, but it's worth it for the times that they do.
And often times when I've changed my views like that, there's a cultural shift a little while later. A lot of what is held as sacred truth is due to fear of the "opposing idea" being right.
Think about programming in the late 90s and early 2000s, but not believing OO to be a panacea. I use OO when it's appropriate, but it's often not. And speaking up about that in those times meant you were viewed as ignorant.
If you program in Java in the year 2000, you have a vested interest in OO adoption. You've invested your time in a language that was explicitly designed to not allow for anything but OO. And if OO is not the end-all be-all of programming, then maybe you've wasted your time. So anyone claiming otherwise is indirectly threatening your livelihood. You're terrified that they might be right, so you insist that they never could be.
You say that but at the same time there's people that voluntarily had holes drilled in their skull.
We agree!
I woke up immediately because that isn’t possible.
It is a strange phenomenon, that the brain tends to ignore things that are expected, and is often jarred by the unexpected. It’s maybe one of the downfalls of expecting the unexpected. Because it makes even the unexpected seem ordinary. Or maybe the reverse is true, and expecting the unexpected makes the ordinary seem extraordinary.
I’m gonna go and have some breakfast.
Yeah this is what I was getting at. Ive been noticing that people have been dissatisfied with the quality of content recently citing marketing type activities like this. Kind of wish it could just be a forum without the competitive drive that we are bombarded with everywhere else.
Joining a team that is scared of technical debt and runs circles round themselves in a bid to avoid it end up producing more debt than they would otherwise. For example, making code overly DRY and wrong abstractions coupling unrelated parts of the system unnecessarily.
We must embrace the reality of technical debt. It is unavoidable and not something to be scared of. It can be managed by keeping things simple, clearly tested and keeping a cool head.
I would say that's just bad engineering due to lack of insight or experience. DRY is the most simple and most dangerous principle (or "best-practice") of them all, because when it goes wrong it's the one that hurts the most.
That's just another form of technical debt. Over-engineering is also a well known cause of debt accumulation.
When you're writing it the second time, you're accruing technical debt. You pay it off when you refactor for the third component.
Compare that to the other kinds of debt (convertible, for instance). They cost a lot of time to obtain and, potentially, a lot of money in the long run if you do well.
Many of the most successful startups had (have?) shitty code. Of course there's great counterexamples (i.e. WhatsApp having a 5 person backend team when they were acquired), but as a programmer and founder I find that my urge to make shitty code and ship fast is waay smaller than my urge to make fantastic beautiful code.
Of course shipping fast does not imply writing crap code in general. At all. But sometimes it does, and when it does, I've learned that hard way that maybe it's better to choose the crap code.
How does needing 40 people instead of 5 to run your infrastructure compares with the carry cost of monetary debt?
If you were only arguing that the people around here probably needs to be more open to technical debt because we here are mostly perfectionists I would probably agree. But I don't think I can agree with most of what is your post.
This is my point exactly. Until you've found product market fit, the size of the infrastructure team "once you're big" should be the least of your concerns.
Personally I don't understand how any tech company with a 5+ person infrastructure team can still be called a startup (unless the infrastructure is the product ofc, eg if you're a PaaS or something)
When I'm a little more aware of technical debt being created - I track and manage it just like a feature or a bug in the project management system. Separate type, label or category of ticket. It surprising several tools don't have this out of the box for some reason.
There's a few benefits to technical Debt being categorized separately:
Tracking technical debt helps provide a sense of the technical debt tickets that are being opened, and floating around both in numbers, size and unknowns.
Keeping technical debt items tracked along side features, and bugs in it's own category is an invaluable form of cross-training to new team members who are learning the why the current code base came to be how it is, and not just what they may see or think is best.
We see how good we might be as a team and individually at identifying technical debt, and if we are improving at identifying it, and help develop a ratio for future accuracy.
Regular technical debt review helps create a sense of what technical debt is occurring, and what can be fixed under the hood on another project, if desired, or possible.
Keeping an eye on technical debt is important too, if something initially small is at risk of painting the project into a corner, or becoming exponentially difficult to re-factor.
Would love to hear strategies and tactics you might use in your TD practice.
We make the technical debt ticket anyways describing the problem, taking a few more minutes to describe "how it should be" so a design process can begin... it will be much easier to pick it up later on... maybe even tie it to an ongoing Technical Debt Roadmap that can be tied into the main product roadmap.
http://www.ontechnicaldebt.com/blog/bad-code-isnt-technical-...
Looking at technical debt in isolation and always regarding it as a bad thing always seemed simplistic to me.
To start with, you don't. Baseball managers and scouts look at a lot of metrics: runs, hits, walks, OBP, VAR, number of pitches in a career, convicted felonies, etc. But at the end of the day, there are people making judgement calls. They're just, ideally, exceptionally well-informed people.
So what I'd like to see, at least initially, is a healthier universe of metrics we can automatically grab from code bases. Deciders can set up dashboards or reports for those as needed. There would still be actual conversations and decisions involving actual humans, but they'd be looking at this data as they're having qualitative discussions about their options.
(relevant, not a complete answer by any stretch)
Articles and handwavy comments tend to dismiss technical debt as a shrivelling pesky geek who keeps waxing on about how vi is better than emacs. (Someone who doesn't have a lot of influence, nor as important).
Pushing techincal debt to be managed by the PO, Scrummaster, busuiness, and/or managers creates major concerns and problems with what you're producing. The passive agressive response: "well convince them" is naive and wrong. They don't care. If we've seen anything from the experiences below, they just end up hoping those days won't come.
Examples:
1. Experian- I'm fairly confident that they had at last one engineer there that bitched about how much the dependencies were falling behind. If they have good ones, there would have been talk about how to automate that process.
2. Half baked products/data loss bugs - (See mongo)
3. Frequent "redesigns of projects" - There are major companies out there that have had to redo their entire product range due to this. It can't possibly be affordable. I'm sure theres lot's of lessions on the migrations. But it's a lot more expensive in the long run. Side rant: I guess if you believe that you're going to go out of business tomorrw, this is justified.
My point:
1. Be a professional developer, get deep knowledge on frameworks
2. Take your time and develop correctly at the start
3. Build experience with engineering, not the latest JS library that ignores experience.
4. Actually give a shit. Venkat Subermanian made a great qoute at one time: Your code quality is a reflection about how you feel about yourself and your coworkers.
Scapegoating the business is the easiest thing to do. But when push comes to shove they just don’t want to put in the work.
On the other hand, it is nightmarish to toil through some tightly coupled mess and try to rewrite it in a manner that is more maintainable.
One makes you feel happy and in control, the other makes you feel like everything is falling apart.
I'm doing that right now, and it's great. There's several responsibilities that are all tightly coupled, and I'm pulling them into multiple components. It's requiring a lot of code archeology and reverse engineering of intent, but it's incredibly satisfying at the end.
The idea of technical debt is that you are trading some kind of completeness/correctness in the code for something else that is valuable. Usually that valuable thing is time, but it can be something else. For example, perhaps you know that your requirements are wrong. However, the client is unhappy with answering questions. You could pester them some more, or you could implement the obviously wrong requirements and wait for them to find the problem themselves. In this way you defer the frustration of your client to a later time when they might be more approachable.
It should go without saying that you should almost never do the above: I'm only using it as an example of tech debt that isn't to do with saving time.
There are lots of times where technical debt is beneficial. The key is to understand the cost/benefit. This is where we usually go wrong. The cost/benefit analysis is usually done by someone who only understands the benefit. It's just making the analysis of borrowing money and making the assumption that you don't have to pay it back. Hmm... how much money should I borrow in that case? Obviously, as much as I can get!
That said, there are definitely early technical decisions that will be very painful to change later on, like how you authenticate users and protect PII.
The metaphor in this article fits perfectly- the developer who wrote this borrowed from the future, and without being there at the time, I can’t really judge their calculations.
As has been said a few times in this thread: there’s nuance to engineering, and a careful trade off. I think the metaphors are weaponized to a point where that nuance is lost in harsh memories of student-era credit card debt.
I've seen the insides of hundred of companies, and the number one cause of technical debt is lack of proper manpower to work ratios. Usually due to short term thinking of IT as a janitorial cost sink instead of as an investment.
any company that can break this mold is going to be light years ahead in the long run, because I have a secret to tell you all...
The infrastructure holding most businesses up is set to crumble at the slightest puff o wind, whether that wind be legal, like a lawsuit or an audit, or technical like an internal data breach...
I found the article excellent and I like the financial debt metaphor a lot. However, the quote above is somewhat confusing. I usually get rid of technical debt (post task) so that a developer can read and understand the code better. They need to do this in order to make some future unknown change. Since a "better design" would at least involve understanding the current design then a strongly disagree with leaving technical debt in code you do not think will be touched again.
I did this for one SAAS company and it worked out great. Although I had been very much a design-everything, pre-optimize excessively type of coder/designer, for this project at a company I co-founded I went completely the opposite direction.
With only a slight nod to the architecture I expected we'd need later, I just coded up the first version in the easiest high-level tools we had, separating only one module. This allowed us to get a much better handle on the actual requirements, especially since the marketing guys of course went out and sold some of the prototype. Fortunately the sales were small enough that we didn't get swamped, but it gave me real data, and strongly reinforced my plan to build a highly modular set of interacting services such that each could run on separate hardware, which could also be scaled in hours. This very scaleable architecture later allowed us to take business from competitors who were (AFAICT) were stuck in a monolithic architecture.
But after the first version had served it's purpose, no one has ever read any of the code again in the 15 yrs since. We just started on the main version and never looked back.
So as a person who formerly would have recommended the opposite, I can strongly recommend making a really brief Plan A prototype / first version that you plan to throw away. Just make sure that you do actually throw it away, and soon (code that sort-of works does have a way of sticking around too long).
Small team size means you don't have a lot of these problems.
this has a lot to do with how your code looks to begin with, bad code is generally hard to refactor, and refactoring is not a cure for bad code.
I'd think it may not apply so much with huge teams, as if you probably need to have a very extensive definition of the problem to justify a large team at the outset, so doing the long-term architecture & design up-front (vs. a throw-away version) would likely be best.
I'd hope for a situation where the initial small team built the throw-away, ran it hard, then use that knowledge to build the big version and the big team.
My greatest accomplishment at my current job has been deleting 45K-60K lines of code.
Not every bit of code will be perfect for many reasons, but please be thoughtful at least, and not just hammer out code. Someone else is going to have to clean up your mess.
I've created loads of technical debt over my career, in the process literally creating whole product lines that later became the career home for teams of people.
So it is more like survivorship bias, you only get to fix debt that is still alive. Debt that went away and no one is ever paying it is nowhere to be seen.
So it's like survivorship bias, where the most important technical debt (the one that kills its project) is often not noticed.