Technical debt is an invisible $1.52T problem
wsj.com
wsj.com
Given that corporate America has decided that all employees are meant to be replaceable and therefore transient, it is not rational for any individual to orient themselves to the long term or make near term concessions. Likewise if employee compensation is a function of hours worked more than it is a function of the company's success, it is irrational for an employee to make decisions with respect to the long term, especially if they will be rewarded for speed over complexity tradeoffs and they can leave the company with a great resume when the debt becomes overwhelming.
Admiral Rickover's speech Doing a Job speaks quite a bit to technical debt and the forces that promote it: https://govleaders.org/rickover.htm
It's good for the economy for new startups to tackle old problems in new and novel ways, throwing out decades of legacy cruft and ossification.
It also puts more comp into the hands of the founders and frontier employees that pave the way rather than funneling it to institutional shareholders of an old business - pension funds, etc.
This risk is somewhat mitigated by demanding only open source software be used. At least then if a software's supplier goes bankrupt then updated versions can still be produced. That's not the case with closed source software.
If your solution starts with "for this to work all software used will have to be open source" you have a completely nom-solution.
Reality doesn't care about your ideals.
Great reference by the way.
This is heresy, in today's tech industry:
> The manager, of course, remains ultimately responsible and must accept the blame if subordinates make mistakes.
> Complex jobs cannot be accomplished effectively with transients. Therefore, a manager must make the work challenging and rewarding so that his people will remain with the organization for many years.
> The ideas I have mentioned are not new—previous generations recognized the value of hard work, attention to detail, personal responsibility, and determination.
Engineering is not a new discipline. Software engineering is, but humans have thousands of years of prior art, and it is being actively ignored by today's tech industry. I was privileged to work for a 100-year-old engineering company. It had a lot of frustrations, but they made damn good hardware. I learned a great deal from them (and also banged my head bloody, trying to get them to let me deal with software differently from hardware).
I think that this should be on a billboard on I-280S:
> ... when the details are ignored, the project fails. No infusion of policy or lofty ideals can then correct the situation.
We did a great job of delivering yesterday's technology, tomorrow.
I wanted them to add flexibility. Things like a different QC system, iterative development, and faster releases of software.
I won't get into details, though. Much as I wasn't happy with things, I had (still have) tremendous respect for the company, and wish them nothing but good luck (they'll need it).
I will say this, though:
I worked with some of the top engineers and scientists in the world. The company had been producing precision optical equipment for 100 years, and people paid a lot of money for it.
It was extremely hard to suggest that they might want to consider doing stuff differently. They had a formula that worked quite well, and not changing on a whim was a big part of that. I think most folks, hereabouts, would go nuts, working that way, but you can't argue with the results.
I learned to write software of a pretty high Quality, as a direct result of working with these folks.
Doesn’t “measure twice” imply some QC is being done before the “cutting”?
They basically had "Quality" as a part of their entire culture. Each person was expected to do their part in the Quality ecosystem.
The spec process could take months; years, even.
Once a spec was written, it became Holy Writ. It could only be changed or amended with buckets of blood.
The projects tended to be about 50% spec, 20% initial development, 30% post QC, and then, 50% fixing the problems found by QC.
We tended to deliver software that people wanted, eighteen months ago. It worked to spec, but didn't do much else.
You're better off just making friends and moving on so as to not jeopardize your reputation or risk your career.
The problem comes when technical debt is never paid off and continually accumulates, even when it's not necessary for it to.
I don't think the name fits at all. It's tech centric, which is self serving of the tech community, by the tech community.
I've found good success in the past with associating my optimizations directly into the cloud bill. This looks like "Improved efficiency of {MyService} by optimizing {cache hit rate}, reducing the number of pods required of the service by 25%, saving {$300k/year} in AWS costs." I look for these types of wins every time I am on a new team.
Candidly everyone in your leadership between you and the CFO/CEO is going to think you just made them look bad.
And you did.
It is easier to keep a bad decision hidden than it is to admit you fucked up, or worse that you burned a pile of money.
I have been consulting over a decade now.
I used to wonder if I was going to get pushed out based on age... I suspect that I could work in tech till the day I die.
* Reducing the number of pods used by a micro service doesn't reduce cost by 25%. For most micro services data storage, data transfer, and other required dependencies will be a very large chunk of the total bill.
* In the organizations I've been in the director is always looking for ways to reduce the cloud bill. This is always a huge line item for them, and the modern microservice architecture is only making the cloud bill higher.
* How is this anything but objectively good for the company?
* Avoiding stirring the pot won't make you any enemies in the workplace, but it won't make you any friends either. Deliver value where you can and the workplace politics will be what they are.
* I've be a part of, and have seen, cloud bills cut by as much as 80%. When things get that bad you won't lose reputation by fixing it because people in many departments are feeling the negative impact of the microservice.
d
Planing seems to be a lost art. The word "capacity" is something on one uses in an agile/cloud based workflow.
If you want to be an effective roadmap participant, learn to phrase technical problems in terms of immediate benefit to customers. It can then at least be prioritized against the reams of other immediate customer benefit ideas in the backlog.
If you want to marshal an impressive development cadence, focus far upstream of 'technical debt'. Whatever that term means to you, it's a lagging indicator. The solution is procedural, not technical.
-Clippy
I've seldom seen it used positively, to a point I don't actually believe it has any benefit as a term and given that I don't think it actually exists.
It's not an actionable term. Talk about and make cases for system performance, operability, functionality, security, cost, support knowledge and similar and you'll have much easier value based conversations that have a chance of having continued buy in long term from your organisation.
Left to their own devices a lot of engineers will go to enormous lengths for either marginal improvements or changes that actively hurt. I can understand why leadership is skeptical of these investments because even for someone extremely technical, if you're not in the weeds everyday of that particular problem, it's very hard to know what's worth it
Putting small tickets that help reduce technical debt, increase developer happiness or help make progress towards a longer term technical goal at the bottom of the backlog because something larger and product focused always gets higher priority is such a great way to ensure you end up with lots of technical debt and unhappy teams of developers.
It is demoralizing to continue to identify problems and outline solutions but they never see the light of day. Also when you encounter the problem again and again and again it's reliving this negative memory of "well this is never going to change". Eventually this happens long enough where it becomes a culture at the company and most folks stop investing their energy into identifying them and things get worse as time goes as you cycle through new developers because people quit and new hires come in. The company loses so much with the loss of domain knowledge and hiring that the product greatly suffers in the end.
Never under estimate how much of a difference a motivated and happy team can make.
While that is true, I don’t think it could be a communication-from-developers problem. Tech debt impairs development speed and causes small irritations in the every-day use of the product.
In my experience, what works is extreme ownership. Someone with agency has to care.
You’ll never get a chance to refactor code if you ask for it. The business folks and users themselves don’t care what the code looks like. They don’t care if there are 11 different micro services and a hodgepodge of pudding holding it together. They never will.
Don’t waste time trying to write a better pitch. You won’t get to halt development to essentially deliver nothing as far as they’re concerned.
Write code that is meant to be deleted. Refactor mercilessly. If code looks like crap but had sufficient tests, rewrite the code. If the tests are crap write better tests. Then rewrite the code.
Don’t ship the first thing that works on your PR/MR. Take some time to compress that code down, get rid of any cruft, and make sure it’s easily replaceable. You never know what new requirements or changes in specs will bring you back to this code in the future. Make it easy to replace (in the worst case scenario) if you needed to.
Wrap code that isn’t under test in a good API. Make the API call into the bad code. Write good tests for the API. Replace the bad code paths as you go.
Part of our job is writing maintainable, stable, and reliable code.
But if they’re deliberately asking you to cut corners, start documenting things.
There’s a time and place for “good enough,” and then there’s straight up neglect. Avoid the latter.
I've been trying to push the phrase "Design for deletion", as a contrast to designing for a flexibility or extensibility.
I used to try to predict far-ahead feature or architectural needs, but more often than not they either never materialized or appeared with some unforseen incompatible quirk.
Instead, I want to ensure that whatever I write can be ruthlessly ripped out and replaced with a minimum of pain.
It's not quite the same as a blanket priority on decoupling code, since statically-checkable compile-time coupling Isn't the big problem, and might even be preferred over runtime plugin architecture astronautics.
This has worked for me since I was taking out the trash at a fast food restaurant all the way to leadership positions at Fortune 50.
One could argue that technical debt is really just regular debt. If can see that, then you also will see that federal fiscal and monetary policy are really the long term fix, not IT departments saying “No” which of course they already do.
Like cost per user for infra? operating cost per hour, by hour? like that...
You need an accountant embedded in your department to just watch and track spend and map it to something. Preferably they are in tech and told to hold product teams feet to the fire.
Where should I put the line items for technical debt, and how should I quantify it?
I've seen plenty of environments where the technical debt stares you in the face wherever you look, but no reports make it to decision makers (and the ones to make decisions are generally not the ones in the position to see all that debt).
If you're in an org pretending everything is fine, that would probably apply to many more problem topics and not just technical debt.
I'm not up to speed on the latest and greatest research, but last time I went into that rabbit hole it boiled down to hiding problems or moving them around so the KPIs look good (be it KPIs in the company, or in the case of publicly traded companies, the 'virtual' KPIs of news and press releases), and the degree to which hierarchy forces everything down.
That directionality gets exponentially worse when you have a local culture of strong hierarchy, and a company culture of making leadership look good (or honour culture?). I've only seen that personally from a slight distance where it really looks like a kind of dictatorship where management comes up with an idea, and the subordinates have to make due with it, and only positive results may be shared or they get fired. The ideas are usually bad or really hard to execute so you see people use all available resources to present 'success' with not much left for the actual thing that needed doing.
There are no repercussions, and no mechanism to correct this. There's also no easy visibility into the fact that this is occurring, aside from the mental strain on junior/mid level engineers.
Even at the principle engineer level, the incentive is to continue collecting paychecks and not rock the boat.
There are many exceptions among technically sophisticated companies, but your average old school company is really a vessel for corporate careerists to build personal wealth.
But all of the above assume that technical debt should be fixed, alot of the time its a huge waste of resources to fix with minimal business value. Most money flowing into a company comes from a few systems, everything else is not worth it.
Maybe it wasn't your intention but it seems like your post is blaming the engineers for the problem rather than the people who allocate the engineers time. The engineering staff is just operating according to the demonstrated core company principles. Of which those poor principles are usually greed.
Also innovation and debt are often intermingled.
SpaceX cutting corners, it could be seen as debt by slow-paced companies, but at the end of the day, the result is here.
https://cc.bingj.com/cache.aspx?d=4904741369235685&w=4jnumCx...
I really like Google's paper on technical debt: https://ieeexplore.ieee.org/document/10109339
I wrote an article with a summary of the key insights in that paper: https://www.ombulabs.com/blog/tech-debt-maturity-model.html
Hopefully, the more articles like this one, the more non-technical managers will finally give a shit about technical debt?
There are trucking companies driving old trucks that break down all the time that are total accounting write-offs. They have competitors with shiny new leased fleets. Both situations are known to accounting and reported on. It's a trade off of what leadership does with that info. Is it better to have no expenses and still bring in occasional revenue or is it better to have reliable revenue with high expenses and low margin? Not always an easy decision but definitely not one that resides in accounting.
No.
Initiative: CREATE MORE TECH DEBT RESOLVING THE TECH DEBT
- Developers are vastly overrated. They keep developing new frameworks when they’d better spend their time solving business problems. Given their cost, no-one dares telling them anything.
- Can’t motivate them to do tech debt.
- They always suggest to refactor everything.
It’s basically impossible to give business sense to a developer. Otherwise they wouldn’t be developers! But they want to be free.
It’s not a “My boss told me not to do tech debt.”
It’s “Every time I’m given free time to tackle tech debt, I open Draw.io to create charts of what I suggest as a new infrastructure with microkubes.”
As an engineer manager, I do agree with the sentiment though. Keeping engineers focused is tough. Allowing them time to burn down tech debt might mean they go off and work on interesting projects instead of something of business value. I’ve also been burned by the “let’s start over and do it right” refactor that resulted in no customer value. Making software is full of tradeoffs.
Quality aspects can be measured, and resulting requirements will mitigate technical debt, unless you put that analysis so far down the backlog that it falls off. But all of that is management or 'team overhead' work.