How to Leverage Technical Debt
medium.com
medium.com
Now consider technical debt. When you go to your architect and say, "I want you to hack something so that we can get it out the door a month earlier", your architect should respond with, "What are you going to do with that time I give you?" If the answer is, "I just think we need to get to market as soon as possible", then you are doing the technical debt equivalent of spending borrowed money on pizza parties.
As a reasonable architect, I need assurances that I'm going to be paid interest on the loan. If I give you a month, I'm going to need a month and a half back. You need to explain to me how you are going to make that kind of time. If you can not, then the business plan needs rethinking. No respectable banker would say, "Oh, there is a 0.1% chance of success if I lend them the money, but a 0% chance of success if I don't. I'd better lend them the money!" He says, "Come back when you have a better business plan".
The other annoying issue is that complexity in software can grow exponentially. So the cost of repaying the debt can quickly outstrip your ability to pay. Imagine borrowing money and having an interest rate of 1% for the first month, doubling every month. Seems like a wonderful idea to borrow the money, but you better pay it back fast.
The problem with technical debt is not that you must avoid it at all cost, it's that you must think very hard about whether or not you can afford it. If you can't, you should avoid it. If that puts your company out of business, then you need a better business plan.
There's something to be said for "just doing it", and so so many times have architects completely gotten in the way of making good progress.
Architects, on the other hand make products. I once was interviewed by a firm that asked me a pointed question about what I did to stop a failed project from failing. My answer was that I gave the best information I could, but that I left it up to the business to decide whether to fail or not. I still think that's the right answer, but it lost me the job.
As architects/programmers, we do not succeed by making money. We succeed by writing software. Several times in my career I have been hired to write shells of programs -- demos that will secure funding, knowing that the underlying software was essentially a lie. It just needed to be impressive and potentially disruptive. Probably the company who would shell out for it would cancel it the first chance they got anyway. Is it the type of software I would like to work on? No (though it did pay well ;-) ).
In any case, that tension is bound to exist. As long as there is good communication and honesty, the tension can provide good results. One of the first questions I ask in a job interview at a start up is, "Do you intend to make a product?" The second question is, "Do you intend to make the product successful?" The questions about technical debt are only really relevant if the answers to those questions is "yes".
The question about what you did to stop a failing project is an odd one and probably completely ineffective. If that's the reason they didn't hire you, take it as a good thing. There's too much nuance for it to be considered useful without other considerations. My recent situation was that I pushed too hard to stop a failing project and was socially cast out. By the time the senior staff noticed that my suggestions actually made sense and weren't batshit extreme, the social dynamic was too much and I left.
On your last point, I'm not sure that would be a useful question, either. It's so incredibly for a linux lead to say, "Our infrastructure security has no technical debt", for example. It's easy to say, but impossible to prove without being at the company for a while, even for an in-the-mud admin. Hell, that's what happened at the place I just left.
One of the most challenging aspects of this in real development jobs is that shrewd managers and/or business and sales workers are very good at deflecting blame and creating political situations in which they are able to pretend to have supported the "right" choice no matter what the outcome.
So, very often, when features get hacked into the system against the architect's advice and it predictably causes untenable debt growth and parts of the system fall over, what is the first thing you hear? That it is the engineers' fault for not building the system correctly.
It's very rare that an architect can successfully communicate the problem of excessive feature growth and convey after a system failure that it was not really the engineers' fault, rather the fault of everyone forcing their poorly conceived features to take priority.
In a healthy development environment, everyone should be hearing "no" very, very often when asking to add features, especially when they come to you as if it is a big emergency that some market phenomenon mandates that the feature must be added right now.
If you find yourself in an environment where your employer does not empower you, as a developer, to say "no" in that situation definitively, then rather than trying to fight any blame wars about whose fault a system failure was, just get your resume ready and leave. There's no need to put up with that sort of poor engineering culture, and plus, wouldn't you rather put your name on something of quality instead?
I much prefer this characterization, where the engineering team are the lender, and they can call the business in to confront them with the unpaid bills that are coming due and demand a structured plan for how they're going to clear down some of their debt.
But there are times when the situation is not perfect, but you still don't want to leave. These can be good times as well. If the project has little or no chance of success, why not experiment with some unconventional techniques? It could be your last chance to do something like that. Working in a group which is failure tolerant can be liberating as well.
Finally, make sure that you fully understand the business decisions before you go overboard. In the very first start up I worked in, the management pushed estimates that were wildly inaccurate. We'd say that it would take 4 months and they would promise 1 month to the customer. We failed every single time. When we finally got to a product launch, many years late, I asked the CEO if he wouldn't have rather known at the beginning how long it would take. He smiled and replied, "No, of course not. I would never have gotten funding if I told people that it would take this long."
There are things that we, as programmers, know very well. There are things that we do very well. This is also true of business people. Sometimes we, as programmers, can't understand what they are doing because we aren't business people. Sometimes in our hubris, we think we know better than them.
Knowing who to trust in these kinds of situations takes time and experience.
It's true that software budget estimations are by their nature educated guesses at best. But we should still make an effort to be honest about them.
Usually the technical debt will accumulate to the point the product can no longer move, is stuck on an outdated stack, and dies.
The dynamics of getting to market are complicated and I don't agree that tech folk undermine them to any special degree. If you're an engineer with a few years experience, chances are you've been smacked by being too conservative or too loose with tech debt. Good contextual use of debt is a key part of the "senior" in any senior engineer. I'm happy to be "sloppy" and fast at the right moments.
My experience has been that business people value time to market, but only if they can offload the costs to engineering by taking on tech debt. When asked to put their money where their mouth is and pay for more engineers or cut scope, often the business side will balk. It's an annoyance when non-technical leadership perennially sees tech debt as a way to get a free lunch. Ain't no free lunches.
For example, I've found that anything related to the general architecture of your app is very important and you should almost never cut corners in those areas - This includes decisions related to what framework you will use, the structure of your URLs/links, how your app will use/reuse components, database engines, etc... These are fundamental engineering decisions and if you rush those, you will likely suffer for many years.
On the other hand, I found that it's generally OK to hack together the contents/features of individual pages within an app - If you mess up one page, you can always rewrite it fairly quickly.
Software is like a tree - If some of the leaves become diseased and start falling off - That's OK; so long as the tree's trunk is healthy and sturdy, the leaves can grow back quickly. If the trunk is rotten and the whole tree falls down - You will have to start again from scratch... If it doesn't crush you on the way down.
Technical debt will be incurred, if you like it or not. Even if you spend a long time on architecture and abstractions, developing software is a process and the "fantastic" architecture you created a year ago is now hopelessly outdated. This happens because over time your insight into the problem domain increases which will give you a better understanding on how to approach the problem which will lead to better decisions on all sorts of things regarding the software you're writing.
Having refactoring as part of a software development team culture can help you manage technical debt. If you assume that the code itself isn't precious but the experience of the developers writing the code is, then you can move forward. You can have refactor sessions at regular intervals, e.g. every Friday afternoon or a whole Friday every fortnight, whatever works for you and your team.
In a refactor session the whole team can work together to make the software better; many eyes make all bugs shallow [1]. It also give an opportunity to create better synergy and banter between team members.
Right on. I've seen discussions about feature toggles which suggested not using them because they led to an explosion of code paths. But that won't happen if you delete the toggle (and the toggled-off code) once the a/b test ends or the feature is rolled out or whatever. It takes discipline and a culture that encourages that by being willing to review / deploy those changes, though.
I think it's interesting to contrast the thrust of the article with Ward Cunningham's original definition of technical debt: https://www.youtube.com/watch?v=pqeJFYwnkjE
Wards definition is much more about the differences between the current implementation of the software and your understanding of the problem, particularly as your understanding changes over time.
He explicitly says that "tehcnical debt" is NOT an excuse to write known-to-be-poor code, which is why I disagree with the //TOBECLEANED approach.
This change in perspective shifts the point of view quite a bit. In some ways this problem defines a startup, your biggest problem is that you don't understand your problem. It's somewhat likely that you won't even have the same problem you'll successfully solve later. The first points are salient here, focus your engineering efforts on the things you don't understand yet, and the minimum support it needs to be viable. Shorten your feedback loop as you navigate the things you do not yet understand.
In this light, a startup is actively SEEKING technical debt, because it represents gaining understanding about a problem, and exploring that space quickly. This doesn't mean you have to "cut corners" and do things you know to be flawed, but it suggests focusing on small 'forays' into the unknown, gaining understanding quickly, and being willing to throw away your exploration for a new one if it isn't working. Then once you find a problem you start to understand and seems like a viable direction, you have a body of 'technical debt' that is the work you need to do to bring the software to your new growing understanding.
It seems like being willing to throw away the exploration if the direction isn't working out is a key advantage of a startup, along with the quick iteration times and lean support for explorations. This is why I dislike the suggestions of writing known-bad code. You're slowing your exploration down, but you aren't gaining understanding of the problem. It's junk debt, not technical debt. You can carefully minimize the supporting infrastructure your exploration requires, and avoid writing extra code, but you can't just slap it together.
It's not enough to flail around building as fast as you can, you have to deliberately learn and explore problems and understanding as you go, and evaluate which ones to put more exploration into and which to abandon. To me, this is the core work of a startup that differentiates it from established companies who are executing on a reasonably well understood problem. The (sometimes gradual) transition happens when you stop exploring, and start focusing on clearing the technical debt from your current exploration.
Thanks for sharing the article, I enjoyed engaging with these ideas!
You want to know what technical debt really is? The result of procrastination in the realm of tech.
It's not even worth giving it a funny name like technical debt that upper management doesn't really get. They understand what procrastination is.
You can't spin procrastination. It is what it is.
Procrastination may lead to technical debt, but it's not the only source. The biggest source of technical debt is insufficient planning for future product requirements, leading to old code blocking new features.
That's why the concept is called "debt," because it's as if you took out a loan on the first iteration of the product, and then many features later, you're still paying interest on that loan.
So no, technical debt is not purely the result of procrastination. Sure, procrastination, especially in the planning stage, can create technical debt. But even the most productive people, those mythical creatures who never procrastinate, are vulnerable to creating technical debt for themselves. In fact you might argue they're more likely to create technical debt, if they choose to skip the planning phase and jump right into coding.
This covers both low prioritization and procrastination, the former of which I think is what you're getting at, and the latter of which I think is what purpled_haze is talking about.
It's the difference between "we'd like to clean up this technical debt but these new features are too pressing" and "we have nothing else going on, we're going to clean up this technical debt..." and then never doing it. In my experience, the latter happens because no one ever generates a real execution plan to get it done, often because there's just no reward for doing so.
A good example of that might be in log refactoring. I recently quit a place for what amounted to a complete lack of interest in correcting bad logs. After spending 8 hours trying to fix some non-prod issues and being completely halted by timeout errors, I was told "Those errors are fine, they even happen in prod (!!!). We talked about fixing it, but agreed to do it way further down the line." So, not really procrastination or technical debt, but something similar since it's 'debt' and 'procrastination' are simply symptoms of a larger problem.
Procrastination is letting for tomorrow what you know you should be doing today.
Technical debt is what happens when you try to do today what is due for tomorrow but you know deep down it cannot be done properly in less than 3 days. Then you deliver and you pray that none will be looking close enough to notice the difference.
What the article says is that technical debt is actually a good thing if whatever you are doing today has more than 50% chance of being thrown out of the window by the end of the month. If is better to try 20 crappy things, discard 15 and keep 5, than to do 8 good things and keep 1 or 2.
But this is only true when people take next month to fix the 5 things that were found to work in the past "sprint". The reason we all hate technical debt is because virtually no one does that.