Technical Debt
martinfowler.com
martinfowler.com
I had a 280MB xml file containing about 120,000 records. I needed a way to parse out a subset of the records (about 15,000) and then put their data into a db.
I was developing on a vps with only 256MB of RAM, so I wanted to avoid memory intensive operations. I'm using ruby, so I started looking into how to 'stream' the xml with ruby in such a way that I could read if it was the type of record I wanted to keep, somehow manipulate and store the data and move to the next record. The more I looked into that strategy, the more complex and ominous it seemed. I just really wanted to find a simpler way that wouldn't require all the apparent tediousness of streaming the xml file (it may seem simple enough, but there are dependencies you have to install, api's you have to read and learn, etc). There were just too many moving pieces and it made me feel eery about the outcome. Another aspect of it was just pure laziness. I just don't care how I get that data into the db - I just want to get it in there.
Fortunately, the glorious wonderfulness of linux utilities saved me from a long tedious solution. Behold:
csplit -q catalog.xml '/<title_index_item>/' '{*}'
find xx* | xargs grep -L 'label="instant"' | xargs rm -f
find . | wc -l
So, two lines of code on the command-line instead of a far-more complex ruby/streaming-xml solution. Now I can have a ruby script process each individual file and add the data to the db - a much simpler problem to deal with.
My point is that you can accumulate code debt by doing the seemingly 'right' solution sometimes. There are probably coders that will cringe that I just used a couple of shell commands to do this rather than write up a long, well-documented, properly OOP, TDD, etc "right" solution. However, the "right" solution in that case would have accumulated code debt - more code to maintain, more moving pieces, more things that can go wrong. I'd trade maintaining two lines of shell code over 10's or 100's of lines of ruby code and it's dependencies any day of the week.
No, I don't think so. This is not technical debt. This is simply the right way to do it. More complicated is NOT better! A lot of people, for some reason especially people who like classes and MVP and templates think that more complicated is better, and will write insane amounts of code to do a very simple thing. I think that's wrong.
The reply is usually something along the lines of it'll be easier to do this and that. But they are not doing this and that, they are doing something simple. Add the complexity when you need it - not in advance.
The video of him talking about the debt metaphor is worth watching, especially the last minute where it gets subtle.
Sure, in any field you can screw up in such a way that it hurts you in the future by damaging your reputation or annoying your customers, or make bad technical investments that turn out to cost you. But in engineering disciplines, doing your core activity (engineering) poorly can hurt your ability to keep doing your core activity (engineering) in the future.
If you're in sales and you cut corners with one client, it doesn't make your phone calls take longer. If you're in a restaurant and you cook things too fast or cut corners on ingredients, it doesn't make your future cooking take longer. If you're a doctor and you're too hasty and mis-diagnose a patient, it doesn't make your future surgeries and checkups take longer. But in software, if you're too hasty adding feature A, it does make your future features take longer to develop. And that's because in software, your future features are built on the previous ones, whereas with sales, cooking, or medicine the duration of any given prospect/customer/patient is relatively short-lived and finite.
That's my best theory as to why it's so hard for non-engineers to understand.
Unfortunately, I ended up wanting to go back to the code to reuse/build on my approaches to the key technical problems... but the hacked mess was tortuous to read or understand (interest payments). So the dilemma is: how to write readable code, when it's only for a prototype and so not worth the effort?
My current tradeoff is to make the prototype code too simple, by cutting out tricky special cases, and by deleting approaches I coded and all my comments discussing them, assessing issues etc. That is, make the code "workmanlike". It might be stupid, incomplete, inefficient and incorrect - but in a crystal clear way.
I actually find this discipline incredibly difficult, and I often lose days at a time exploring alternative coding. This often pays off; but the biggest payoff comes from having a clear and concrete basis to build on. I've just finished the current prototype, and many solutions to technical issues have now clicked into place as obvious to me - because I can now see them. So that works well.
And I also have something - incomplete, buggy, inefficient etc - that I can play with as a user. Ugh. In other words: a crystal clear sense of what needs to be improved. Maybe..."auditable"?
• knows the exact location of the hacks.
• Is aware of the repercussions and is assured that the system can get away with the kludges for the time being without any hindrance in the system’s effectiveness to get things done.
• Knows how to fix them if need be.
• Communicate the situation with others, including the fellow programmers and the managers.
Even after reading the article, I think that folks often don't understand that Technical Debt is not a "bad thing" but a thing that must be understood and managed.
When you understand this, it puts software engineering into a very clear perspective.
That's true, but how do you manage it? Especially when it's so difficult to measure in the first place.
One idea that popped into my head would be to keep a running tab of technical debt that's being accumulated in a project. It's tough though, because developers who are in "accumulating debt" mode tend to be the ones with the least amount of time to spend on upkeeping a document that tracks their debt.
It also gives you a list of alternative things to work on when you're getting burned out on the active tasks.
Very roughly, they are a measure of known technical debt.
It's always nice when I come across a note that's been made irrelevant by other parallel improvements along the same theme -- even though the note wasn't the impetus for the other changes. It's like finding a $20 on the street!
You can't manage this with documents for the same reason you can't manage thinking with documents.
If I were going to try anyway, and were working with a team of more than 2 or 3, I might put a big piece of paper on the wall and ask everyone to write on it whenever they notice technical debt. The bigger the issue, the bigger the note. Maybe use different colors for different parts of the system. After a while, patterns would emerge. But the real purpose of this would not be to track accumulated debt; it would be a device to encourage people to talk about it and think about it.
Is it unprofessional to allow managerial deadline pressure to influence code quality ?? At what point is it justified for a programmer to say "No"? At what point is it wise?
Actually, yes, I think it often is. If you're paying me to write code, then it's incongruent for you not to rely on my judgment about how to do it. If you don't like the results I produce, you're well within your rights to find somebody else. But you are not within your rights to hire me and then block my judgment about what the system needs.
The vast majority of the time, it's not a matter of explicitly saying "No": if you even get into that conversation in the first place, it's probably game over. Asking for permission to do a good job is likely to meet with, "No. Do a faster and cheaper job instead." That's an argument the programmer will never win. And rightly so, one might say! - because of the lack of self-respect they're demonstrating by such behavior. One thing I've learned over the years is that it's my own fault if I don't stand by what I value and know. (Well, partly learned, at least.)
If you want to be professional, don't act like a servant.
Edit: I was thinking about why this is by and large so poorly understood by software developers. I think part of it is that programmers tend to be younger. They haven't had time to learn this the hard way, and there is a large authority and status gap between them and their (usually older) managers. Perhaps another aspect is that many programmers are introverts and thus less likely to advance their own interests and judgments, even when they're right.
I find this interesting because those are both reasons why the hacker renaissance and the corresponding startup explosion are really good things. When the hackers own the product, the situation we're talking about is much less likely to arise.
Other than that, I agree with you. Last year, I heard a talk given by two co-founders (one technical, one not), who were discussing their working relationship. The tech guy got a big laugh with this line (paraphrased):
"Every once in a while, [my cofounder] gets on my nerves with questions like 'why can't it be done sooner?' It's really helpful to be able to answer 'because fuck you, that's why.'"
That's a luxury that most programmers get to experience, unfortunately.
I had seen a Martin Fowler article on technical debt before but the link to Ward Cunningham is new at the least.
As a part-time economist, it occurs to me that the debt metaphor actually doesn't work on some levels: Companies constantly increase their monetary debt, since THAT can be paid through increased profits. You actually wouldn't want a company that grew without increasing debt - that would be wasted resources. Technical debt is much worse to accumulate since it can only be paid by increased development time.
Also, one accumulates technical debt whether one wishes to or not. This is because often the only way to understand how to do a system right is to first do it wrong.
Debt grows logarithmically, if the complexity of your code base is growing logarithmically you're doing great.
The problem is software complexity grows exponentially.
And that's why you can't manage it like you can manage high interest debt.
You can try to manage it, but I have yet to see that management succeed.
Every single time a company has tried to manage technical debt, the debt simply goes unpaid.
How many people here who've worked on a sizable and mature code base can name temp files and variables that have turned absolutely permanent over the years?
That's why software complexity, in large organizations, should be approached like an infestation. Erase and eradicate on sight, don't try to save some for later.
The whole series is definitely worth reading.
It's also a good tool to allow the management to do what management does best: take the technical information and make management/financial decisions based on it, on how much investment can be made now.
It turns a binary either/or choice: devs saying "Our code is bad, we need to rewrite" into a quantitative one: "How much technical debt do we have? Is it increasing? Can we reduce it?".
This guy writes books on UML.