Technical Debt 101
medium.com
medium.com
On the other hand, if you focus on quality, you end up making choices that enable fast development. You make sure you have high-quality requirements and a deep business domain insight in the whole team, which means you end up writing only that code which you truly need, and write it in the right order (most important features first). You make sure you have code that is well-covered by tests, which as a happy side-effect makes it easy to adapt to new requirements. You focus on having a high-quality deployment process, so the amount of customer issues that result from a broken deployment go down reducing the load on your dev and support teams.
Focusing on quality first is not about gold-plating code, it's about making conscious decisions to have a high-quality process from front to back, and those decisions typically result in a highly efficient organization. It's no accident that those teams which manage to ship releases every few weeks (e.g. firefox, chrome, facebook) have world-class products. You have to focus on quality to go fast.
This isn't just theory. Capers Jones actually measured this. The fastest delivery schedules were produced by teams with the highest quality product. http://www.parasoft.com/products/article.jsp?articleId=2769
I'm convinced a lot of developers have been completely brainwashed by the Business in believing they don't need to ship good code. They absolutely do, and the cost of it is high. The cost of maintaining bad code is much, much higher, however. Abdicating this responsibility reflects poorly on you, the developer.
Much of the populism in software targets the insecurity of those who don't push themselves to ship great code: "getting things right is so hard! Just do the best you can and don't sweat those edge cases too much!" I sweat the edge cases when I'm writing the code in the first place because I'm incredibly lazy: I know the mental cost of re-creating all the context necessary to solve the problem later is very high, and I'd rather not return to the feature ever again.
That's one of my favorite sayings.
>I sweat the edge cases when I'm writing the code in the first place because I'm incredibly lazy
I like to front-load any problem solving and make sure I understand what I am getting myself into. What that looks like in practice is that, when designing architecture, I will write up a list of every scenario I can think of (within reason of course, also including future plans), and verbally test each scenario against my ideas. Essentially I try to build up the largest image I can of all the moving parts, and then narrow the scope back down to the present problem. Initially expanding the scope to include edge cases and potentially fictitious scenarios not only helps me isolate concerns, but I consequently understand the limitations of my implementation (which of those scenarios failed my verbal testing) and I have my notes for when/if I return. It's how I sate my natural desire to be overly-comprehensive without actually over-engineering everything; I am confident that what I build today is flexible and extensible enough to grow as it needs to. Sometimes I open a file I haven't opened in ages and I read a comment saying, "You're welcome to my future self", and I can't help but be thankful.
I think after about 5-6 years of coding was when I came to the realization that everything you build 'the best way' will become awful coding practice in a few short years. But that writing things at blazing speed just to hack it together and get to work invariably leads to code which is written to work a single way and is a nightmare to maintain.
I try to build small, flexible solutions which are as adaptable and maintainable as I can, with the awareness that whatever I build will be looked as as archaic in a few years, but it might still need to be maintained and upgraded.
So, slow down a little, think a little, code a little slower, but don't spend 6 months planning the perfect code weapon either.
Disagree.
Good architecture is almost timeless. It isn't always en vogue, but it is recognizable by being pleasant to maintain, conceptually cohesive, and simple. It usually resembles a domain model, with the UI, persistence, and everything else being built around it.
It is a force at least as strong as the language used.
Never dismiss the dangers of over-engineering. While not as terrifying as under-engineering, how useful is a feature that exists only for the purity of design?
However, over-engineering and spending quality time thinking about the design and domain are two separate things and typically the more you think about the design and domain the less likely you are to over-engineer.
Developers tend to want to generalise everything because it makes it more interesting and has the appearance of good design. But generalising in one direction reduces flexibility in other directions. Better to copy and paste a bit at first and wait a bit before adding that extra extraction layer that generalises Foo.
Very few workplaces will let you work hard for 5 hours solving problems in the most time effective way and then spend the next 3 hours on your personal development.
On the other hand you can always spread the work over 8 hours and "experiment" with "on the job learning" leaving all sorts of sub-optimal solutions in the project. Then the next guy goes "Why on earth did Jim use X here?" and the truthful answer is because Jim wanted to learn more about X.
Over-engineered components will work (sometimes flawlessly), so you don't realize you've spent too much time and money on something until long after you built it.
;)
The tried-and-true implementation has a known initial cost and cost scaling factor as it's repeated throughout an application, while an experimental (and overdesigned) one is extremely sensitive to context and could become a net negative if it fails.
When we talk about "technical debt," I think we're describing something that isn't an experimental design, it's just "below" the known standard - it's faster to implement, but it cannot scale, and we already know that. Conversely, a standard practice is one that works and doesn't usually get pushed to its tolerance limits.
The corollary of this is that for any engineering project taking on a big enough problem, you have to assume some technical debt to ground the system in a place where each module can be tested as part of a whole: otherwise your work will go too deep into the experimental/overdesigned zone before it ever completely functions.
I would much rather come across something under engineered than over engineered.
It was considered good architecture to keep everything together. Many established and well-respected developers argued against relational data, because they said 'it splinters all of the data' and modular programming 'splits up the application logic all over God-knows-where'.
Some people still argue if Linux and it's monolithic kernel is better or worse than Windows modular design. These are two completely different architectural schools, and yet both have people who consider one good design, and the other flawed beyond repair. It doesn't matter which, take your pick, they both have zealots and detractors.
So, saying 'good architecture is timeless' is kind of general for me.
Things you consider to be absolute today will change and be defunct in five years.
If you're into small modules, stay consistent and separate things. If you're into OO-design, try to encapsulate and make your objects re-usable.
When you start losing this consistency and drift away from the architectural vision in order to make quick-wins ("lets just tuck this feature to this module even though it might not really belong there..."), that's when things turn into mud.
Understanding the architecture and its philosophy gives you a big insight into the code, the way its structured, where things fit together. It saves you time.
With solid architecture you know where to change things and where to add them. I think it's timeless in this sense. Its consistency applies everywhere. Once you 'get it', and stick to it, things make sense.
As long as nothing changes, I can agree with you. But that's the point of architecture in the first place, to be there for you when things change. The focus of architects should be to ensure that changes are as easy as possible and not to have an elegant design. I've seen some of the most elegant designs that are easy to maintain trashed due to changes in user requirements or changes to the market place.
For this reason I advocate, and have advocated since I started developing in the early 90s, micro service/modular architectures. Java OSGi is currently my favorite environment (though I'm also investigating similar approaches in Clojure), but I've done this with C/C++ in the past as well using shared objects/dlls as the modularity mechanism.
Good architectures adapt to change, they doesn't stay the same with time.
Dynamic loading is indeed the ultimate modularity. Combined with this microkernel approach, extensions are just as powerful as built-in services. The only problem is you enter a new hell: managing dependencies/versions of the core vs the extensions. If you can control the release of the extensions as well as the core, you're set.
We have things that are 6+ months old without any of the normal best practices and are able to continue to ship w/o delay.
Hell, for a new project/service two of my coworkers completely bypassed version control and unit tests.
You can achieve high velocity without high quality. The cost to do that is sometimes you have an expensive production bug to fix.
Please note, I'm not endorsing the way my coworkers do things but I've given up fighting any of it outside of my narrow slice of turf. ;)
I was pointing this out not to say it was a good thing but rather that the premise of [Quality == Speed] is flawed.
Making small incremental changes with extensive manual testing can work. It just looks like you don't have to pay for the test effort.
I've seen it a fair bit with internal projects at larger companies but most external customers I have met in my career would quickly switch products.
Edit: It also sounds like you are confusing best practices with good code. I've met teams, usually without a formal cs education, who haven't used version control and didn't have any unit tests, but still had a well designed codebase. Of course best practices are best practices for a reason....
Its an ecommerce site. So its all external customers [including API integrations with external IT departments]. I'm amazed we get away with it tbh.
1) They both have degrees in comp sci.
2) For very narrow definitions of the word "good" you would be correct. Its technically functional and reliable. However, it regularly has problems with security and/or changes to the common library breaking other people's services.
I mostly interface with outside developers for integration purposes and I never really had a normal on boarding process.
I'm just a random developer with no authority/power.
That said, my best guess is:
> 1) What is on-boarding new employees like with that system? They get an intranet project that is relatively harmless and helps them learn the common library -> they get assigned projects on the main site and get progressively larger features to implement. There is no real documentation beyond the actual codebase and asking people questions.
> 2) How long does it take you to get people up to speed and being productive? No clue. I'd guess 2 months based on how quickly I switched from "intranet projects" to integrating Amazon MWS, 3rd parties, etc.
> 3) I would suppose the lack of rules makes them fast in the beginning and in the end they are contributing to the mess in their own special way, doubling down on the technical debt.
More or less, yes. An example would be we now have 3 partial versions of the same intranet application. The super old one, the one I built before handing it off to the new guy, and the new guy's version [because he didn't like my version].
But if no one can understand how the system works and predict the effects of changes, the thing is doomed.
We're a typical "launch in 3 months!" start-up that is now 7 years old, and we've were so focused on growth and feature development to meet the market that technical debt just accumulated and accumulated.
Eventually, technical debt began to have a real impact on business feature delivery, so we talked about it with our business team and came to an agreement that 30% of development time was ours to reduce this major risk.
The end result is that over time, as quality improved, our business feature delivery has gotten better and faster - less bugs, and more features per sprint. We're very lucky to have our business team, I think.
Using the user roles example given in the article, it's not beyond the wit of man to foresee that you're going to need user roles at some point in the future, so you can make provisions for adding it in later. Done right, the extra effort is minimal but means that you'll be able to plug in the new functionality at a later date without needing to refactor the entire codebase.
It might be as simple as figuring out where the permissions/role check will go and including a "# Future permissions check will go here" comment. Or you might actually include a check that calls a dummy service/function which always responds to a "Can this $user do $action to $resource?" query with "Yes". Then, in a few months, you can simply replace the dummy service/function with a real one.
You don't even have to design/architect the whole thing up front. You just need to avoid painting yourself into a corner. Ask yourself "What am I likely to need in the future and can I do something cheap now that will save me a lot of time/effort further down the line?"
If you ever need to explain this concept to a non-technical manager, tell them that it's like applying a kind of net present value calculation to development time/resources: Spending a little extra now means you can potentially save a lot further down the line.
Very good point. I was thinking about this issue not too long ago. And the ability to do that, comes with experience and often separates normal programmers from much better programmers.
It also takes a while to detect this, because there needs to be enough "Futures" that become present to tell if the original architect predicted or foresaw or did they just over-abstract or and overloaded the code with potential apis or features.
More often than not I see the later. "This code never runs","why is this here","we have 4 levels of abstraction for this, why?". "Oh because I thought we'd have this and that in the future so I added in".
Quite rarely I see the former. When a developer is asked "can we add this?" they come back and say "yes easily, I knew you'd come and ask for this so I added the abstraction to handle it or add stuff to it later". The one thing I noticed is that this becomes a pattern for people. Some people create the former type software some the later.
If there is some more meaningful distinction here that I'm missing, can someone clarify?
But maybe a better way to think about it is "leverage" vs "debt". Leverage is something you use to improve the product with a known and (hopefully) quantifiable return, whereas plain old debt represents those poor non-decisions.
In fact in my career I've never seen bad code + good tests unless the tests were added at a later date (typically by another developer trying to do what they can to stop code rot).
On the other hand it is a joy and straightforward to add unit tests to good code and typically they are much less brittle.
Technical Debt is a very specific metaphor that Ward Cunningham framed as a strategy. Ever since, though, people have used the phrase for all sorts of entropy in software. If you look around the literature, you'll see people talking about type 1 and type 2 technical debt.
In my experience (and I'm the guy who wrote the Legacy Code book mentioned in the article), the typical entropy thing that we see in software has more to do with just taking the easy way and avoiding refactoring - different from Ward's concept.
The old, and unfair, characterization of programmers as smelly trolls that can't handle client interactions is more of a crutch for fools than a reality.
The truth is my clients complain bitterly about how they'd prefer to just talk directly to the dev's and not be bothered by the various hand waivers trying to justify their line item on an invoice. Often this is a great way to annoy the dev team, but it's a good expression of the perceived value of a non technical intermediary.
In the end the old truism of "You can't manage what you don't understand" still applies to this new fangled industry of software development.
That being said, ramping dev's up on management techniques is another challenge, for another internet rant.
That was an emphatic statement in the article and I feel the discussion would really be constructive if the author (or posters in this thread) listed concrete examples of startups that failed primarily because of software quality.
If success has many fathers, failures are not orphans and also have many fathers as well. Perhaps software quality was not the 1st but actually the 2nd or 3rd reason why the company failed. I'm not saying such examples of software execution killing the company don't exist. I just can't think of any prominent examples at the moment.
I'm leaving out poor software examples such as the Atari 2600 E.T Extra Terrestrial game cartridge because that was not a startup situation.[1]
[1]http://en.wikipedia.org/wiki/E.T._the_Extra-Terrestrial_(vid...
On the other hand, an internet company like Friendster looks like an interesting case study. Their infrastructure software was not efficiently traversing their social graph. It was something Facebook programmers conquered but Friendster did not. But was software quality the #1 reason behind Friendster's demise? Apparently, the board-of-directors didn't prioritize and throw money at the engineering challenges that Friendster was facing.[2] So the their failure can be reinterpreted as a combination of management of priorities and software execution. Maybe if they did throw money at the issue, it wouldn't have mattered because their programming staff didn't have the unique competencies to tackle at-scale social graph problems. In that case, it's not a "technical debt" issue but a "hiring talent" issue.
[2]http://highscalability.com/friendster-lost-lead-because-fail...
Another case study might be the Chandler software (chronicled in the book "Dreaming in Code"). However, it appears that the product got bogged down by architecture astronauts unable to make decisions and not "technical debt".
If there are thousands of companies that failed on software quality, it'd be interesting to see which ones can be disentangled from non-software issues and we can confidently say, "yep, that technical debt is what killed them."
(My curiosity for real examples does not mean I'm endorsing technical debt. I'd just like to get some real-world perspective on this topic.)
As programmers We build code paths. Design mazes. In the beginning the maze is simple, there's only a few paths. To make things simple we separate our ideas into many small mazes. Since they're small, it takes a short amount of time and concentration to predict the different possible paths in our heads.
Technical debt is when, due to to time constraints the decision is made that it is quicker to draw one more line on an existing maze, rather than create a whole new small maze. In the beginning it seems not so bad, you can still take in the whole maze at once. But as time goes on, that one maze gets larger and larger, and finding a successful path through it takes longer, and longer. Eventually you get to the point where you can't separate the maze because too many of the paths are built on top of each other, but you also can't manage the maze because its to large itself. That's the default. That's the rewrite.
I think the real problem of this piece is that it flatters nontechnical managers, as businesspeople, by describing technical debt as a sensible business decision they can get into consciously through clearheaded negotiation with their technical team; then characterizes 'bad code' as something different, that happens when the technical team just don't do their job. This alleviates the nontechnical manager of any responsibility for bad code. The fact is that knowing about technical debt is not necessary to find yourself up to your neck in it. A team which goes for years taking shortcuts, being forced to hit unrealistic deadlines, or which lacks the skills to really build a good, robust system, is like a kid with a credit card who doesn't get that just paying the minimum payment each month isn't going to work forever. Every shortcut they get asked to take, they swipe the card, and they get to walk out of the store with the goods, and it seems to just keep on working; thousands of dollars worth of stuff, and they just have to pay a few hundred each month. It's magic!
But the consequences will show up in the end, either in the form of a long period of hard graft to pay it down, or declaring bankruptcy and abandoning the codebase.
That's why I love the technical debt analogy - different people manage financial debt differently, and most businesspeople understand the concept that planned debt is necessary and beneficial for growth; arguing that Ward Cunningham and the author don't think you should call other technical debt by that name seems petty, and not particularly useful, when extending the analogy to runaway debts works so well.
Overall the problem I really have with this piece is that if I tell a manager "This project is carrying a bunch of technical debt," now they'll google it, find this, and come back to me with "You mean it's a pile of bad code." Which is true, but how does that help us? When I make the argument that a troubled project needs to pay down some technical debt, I'm trying to make clear that we need to start paying off more than just the minimum payment every month. If the technical debt analogy is removed from the table, I'm left with just saying the team needs to replace the bad code with good code. And then they'll tell me that this piece also told them that the 'big rewrite' was a bad idea too... So how has this helped?
Then they start to understand.
When someone talks about taking on debt, it is critical to know the interest rate charged. Also very important are whether the debt is collateralized (example: car loan, house loan) or non-collateralized (credit card), along with how easily it would be to get out from under the debt if you cannot pay (can it be discharged in bankruptcy? how does it affect your credit rating?)
Very few (if any) financial people and senior managers could make the company liable to replay this debt unless these variables were known with relative certainty. To take on debt without this information could be deemed reckless and may expose the company to shareholder lawsuits, etc, and be job-ending moves for the people involved.
Back to technical debt? What kind of debt is it? Is it collateralized? What if you can't pay it? Can you write it off, or do you lose the company? Can you get out of it in bankruptcy? What's the interest rate?
I argue that "technical debt" is like an unknown debt. The interest rate is unknown, it is collateralized with the organization itself, and cannot be discharged in bankruptcy, short of bankrupting the entire company.
The prudent and reasonable thing to do would be to not take on the debt at all. Remember what Toyoda (the founder of Toyota used to say): Cash is your best friend, but debt is your worse enemy.
Who would borrow money to go to college and get an education at 40% interest rate a year? It would only be worth it in very rare cases. What about 20% a year? My wife put her college education on credit cards, paying foreign tuition rates, and we had fun paying it back, let me tell you. When I went to college, I went slower, paying as I went, but had no debt at the end.
If any company takes on debt, it is irresponsible for them to not do a thorough financial analysis of the impact of the debt.
What's the interest rate on technical debt? Who can say?
> it’s something that you get when you delay design decisions for later
But then presents us with a problem where we can "spend" 21 story points now, knowing we won't have to do much refactoring later. How do we know this? I thought the whole point was we couldn't see enough of the picture to make a stable design.
Instead of talking about how patches, hacks, and workarounds impact code, I'd like to see discussion on how you move forward when you know the requirements are going to change, but don't know how.
While still technical debt it is not the type the article is talking about or using in the example.
FTA: "At this point the developers realize that the code is starting to get messy and the solution is refactoring it to have a decent permission system."
I.e. You are at the point where you realise you need a better design and know the rough solution. Do you keep putting band aids on your solution or do you properly refactor?
I've found requirements do not change nearly as often as you would think. Only maybe 10% of the changes I make come from a requirements change.
Take for instance the articles permissions example. Is that a change in requirements or a lack of understanding the requirements and domain?
Sometimes it is a genuine requirements change. Maybe the company has recently adopted a new security policy, or maybe you are selling to a new customer and they require permissions even though none of your existing customers do etc. More often than not though the requirement has always existed but the dev team didn't know about it or didn't have time to deal with it.
Requirements come and go for me. And losing requirements can be just as disruptive as getting them. Worried about how component X will handle users? Well, we don't ship component X anymore, so any refactoring work you've done is wasted.
I might be the odd one out with requirements changing all the time, but I don't get the choice between good design and fast design. I have to creep forward in my architecture, like I'm fumbling around in the dark, because what my system should do today may not hold until tomorrow.
And that is simply the case with certain lines of work.
To help me determine which requirements changes are simply a failure in the requirements process I ask myself:
"If I had shipped feature X a month ago would it have had business value?"
If the answer is no then I'm likely dealing with a legitimate requirements change.
For example: If I worked in the tax industry and the government changes tax laws then shipping these changes months before would not deliver any business value.
Or if I have to make a requirements change because of an external dependency changing. No extra value.
Take your tax-software example. You need to start work before the law is written, but there could be changes in the law that require a rework of business logic. What do you work on? How do you prioritize? And how do you deal with that subsystem that no longer has a reason to exist?
In my domain, technical debt is not really a choice I make. It is something that happens as the system ages. I wish I could get someone to speak to that.
Do you have a rules engine already or not?
If you're hodge-podging it all together and all the rules are implicitly defined by the code as it exists you're in a bad spot. Knowing that the rules will change every year your goal isn't to grind out this year's changes really hard for six months (and then even harder for another moth to change the final changes) and then slack off for five months.
Instead you should implement a system such that the specifics of the changes are captured in the rules. Once the new drafts come out you need to look for what all the possible new rule categories are, what additional data needs to be captured, etc. But then once things are finalized it's a matter of tweaking the rules slightly to capture the final wording of the bill.
This may not be something that applies to you, though. What area do you work in? How does the debt pile up without choices being made?
There are plenty of places that follow along your example: I can build a general solution that will withstand a wide range of changes. This doesn't change the fact that a change could come along that I didn't anticipate. What I would call "debt" is more the result of changes in requirements than choices the team made.
You can still frame dealing with change as classic "technical debt", except the full cost/benefit of each design can't be well known. Instead it looks more like "technical investment"; you try make a design that will pay off, but you don't have a guarantee that it will.
I find myself hedging a lot of my designs, and rarely am I able to justify spending a lot of time up front on any architecture.
The idea is to choose a new design that scales better. If you have M permissions and N permission checks then each new requirement might cause M * N changes to the codebase.
If you have a reasonable permission system you might end up having to make N changes to the codebase (worst-case) or perhaps only M modifications to permissions or perhaps both. But in that case it's M + N changes, rather than M * N. For any values of M and N greater than 2 (and very nearly all of them will be) M + N is smaller than M * N.
But, what if I never get a second group that needs this? While hard-coding values was an ugly shortcut, I may never have a need to revisit the code. If I had gone with a generalized solution, I'd still have to contend with maintaining all that code.
If you ask someone who knows about the business rules if something could EVER happen and they say "no" they're probably wrong, but not maliciously so. Be cautious when taking their word for something. Even if the CEO signs off on the idea of "such and such never needs to be able to do x, y or z" the second a big contract does need it, it all goes out the window.
As a result whenever possible choose solutions that give you SOME flexibility without necessarily creating a supremely robust, exhaustive module.
In this particular case, once you realize that you're going to have more in the way of permissions other than global can/can't do, it's time to be smart. Being smart might only mean creating a function which takes some context like userid and "permissions token" and/or the URL that the person is visiting and then doing a hardcoded permissions check for say one of a dozen userids who are basically root versus everyone else who isn't.
Yes you didn't make a giant scalable system of awesomeness including a web-based permission editor. But you did make a function you can insert every time a new permission item gets created and eventually a system could be created for storing those in the database such that users can eventually be mapped to permissions also in the database.
So if you want to hard-code those values into a single page, which then grows into two pages, which then grows into four pages which then grows into 29 pages that's the wrong solution. But if you want to hardcode those into a function which you then call, that's a good enough solution for now that doesn't hobble you later. Putting the function call in adds very minimal overhead in terms of execution and ranges from slightly negative to moderately positive on maintainability.
Parts of software development are similar to being a speculator in a financial market. You need to have a fairly intuitive grasp of what's going to be a "good bet" for spending a little more time now which has the potential to pay off big in the future.
Technical debt is a good example of a bad analogy :
1. Assume some piece of code works, technical debt is only an issue if that code ever needs to be modified. Financial debt is always an issue regardless of what happens, even if you die.
2. Writing some code that generates the so called technical debt will often have an upfront cost saving, so whilst it might incur some extra work down the track, the extra work isn't a straight cost. The cost = (extra work - time saved up front).
3. Attempting to avoid technical debt often ends up with over-engineered code.
4. Being dogmatic and avoiding technical debt in all projects is often not good for the project / business / investors. Being early to market at the cost of code quality is often very important.
I think point 3 is the worst scourge of all. It is quite common amongst mid-range developers. After working for a while they start to get good development skills and start to think too much, looking for ways to express their skills.
In summary, it is like most things in life, there is a balance to be found.
As a native English speaker and someone who prides themselves on their ability to clearly convey complex ideas through prose, I don't recall a point where a grammatical error detracted from my understanding.
Extreme examples will be floated, but in practice what people consider "bad code" are things that made decisions that aren't sustainable. For instance a complete lack of permissions. A lack of logging. A lack of multithreading if it's a CPU bound process. Locks. Dependencies. Etc.
Each person and team really does define bad code differently, and often in complete contrast with each other. There are people who considered DI heavy code bad code. Heavily abstracted and layered Java code is considered great code by some, horrendous code by others. And on and on.