How a startup can survive technical debt
andreschweighofer.com
andreschweighofer.com
It sounds obvious, but I hadn't thought of it that way. Once I realized I had full control of when to pay back tech debt, it made me more comfortable accruing it strategically.
As an employee, it wasn't my call when to pay down the back-end's debt. I was very happy to get out of that.
Anyway. Yes, it's different when it's your debt, and what makes it your debt is then whether you're the one who decides when to pay it.
This is true but you will always feel that paying off the debt isn't growing the business or giving any real benefit to the customer. If you can fight that then it works. If you can't then you'll get more and more technical debt.
Generally there should be a motivation behind "fixing" technical debt that is expressible in terms of business value, whether that means faster page loads, more uptime, or ability to deliver X,y,z faster (or at all in extreme cases)
If you can't express the trade-off of solving the debt in terms of business value then it's going to be low priority by definition.
Another strategy for things that as engineers we know will make things better in the long run but show little immediate value can be to ask for a small amount of time per sprint/week to invest - ideally you should still be able to articulate why something is worth changing though
Even if there might be some exceptions for some technical debt, it's always true from the product owner's and business perspective.
Keep in mind that it takes resources, and sometimes even downtime, to address technical debt. In the POV of a product manager, addressing technical debt is an investment with the promise of little short-term returns with the tradeoff of significant risk in the form of downtime and regressions and bugs.
And more importantly, technical debt is not a "one and done" deal. There are always more rewrites and refactoring to be done, and the one you're doing today might be the topic of technical debt discussions in the future.
This statement tend to be true when you accumulated so much debt that you cannot fix it anymore, but if you end up there with the premise it's your startup, you failed as the tech lead
They can make things work, but they don't have the experience to avoid tech debt even in the short term.
I have seen this as a CTO, unfortunately I came too late and the company broke.
The terrible management played a bigger part, but the code base was so terrible and the tech debt so big they couldn't ship even very simple features in a timely manner. Testing and release was insanely slow.
It's not even necessarily their fault; they just need piles of time and mentorship and direction that you can't afford to give yet. Worse, you'll quite possibly end up firing them when you eventually start reigning them in and they don't like being told how to build things.
Trust me, it's been an extremely painful experience.
Had this exact thing happen at my company. It's tough when you have to start doing this, but even tougher if you don't. There are times I certainly think we'd have the same output with half the people if we'd kept the bar higher.
The startup I'm currently working for has the same story. A junior wrote most of the code, but we got hired at the first round. It will take months to solve the spaghetti, but it might work, but I'm very experienced handling even huge and terrible code bases (300-500 thousand of lines of legacy code).
The hardest part is explaining to the owners why it's so expensive to fix everything. They don't understand how it was so cheap to build yet so expensive to maintain.
But a team bigger than 3 developers where no one has any solid experience is a recipe for disaster IMO
- Junior: somebody who is happy that they found a working solution. Most code what they write doesn't even work or they don't know yet how to approach problems.
- Normal / mid-level: developers who can easily find working solutions to problems, but not always the best or easy to understand. Should be able to handle edge cases and errors properly.
- Senior: How to solve problems is not even a question for them, they can come up with multiple ways solving the same problem, but their chosen solution will be flexible, easy to understand (even for Juniors) and maintainable in the long term. When they make a technical debt, it is a conscious decision and not a fatal mistake.
With the exact outcome that you describe.
Senior v. Junior is not the problem here. Developing according to your stage and future is. Build things that can be thrown away next week when your product shifts. (ie. low coupling, simple, avoid abstraction adventures).
_Maybe_ "senior" devs are more likely to understand that. I'd argue it's rather devs with any multi-year experience at an early stage startup, regardless of skill level.
Junior developers just can't write clear, simple code. They accidentally over-complicate everything, in the most unhelpful way possible. If you have a senior developer that does that, I would consider them a mid-level developer at best.
I've seen this play out in 2 large companies I've contracted for so far.
At the start of a new project, recruitment look for developers with the lowest daily rate they can find and end up with a team of recent grads / people switching careers within the company etc. That's all fine as long as you add a good measure of developers with >5 yrs relevant experience to the mix. But as these people are twice as expensive, they often start without them.
A few years later the product "almost" works. But it's full of bugs, it takes months instead of days to make minor changes, and it still requires an enormous team of people who are all irreplaceable (they possess the "secret" knowledge about the product that usually would be embodied in clear code intentions, past jira tickets, commit history, basic documentation etc.)
Meanwhile a new competitor in the market has developed a much better product.
This situation can last for years (because some companies are a bottomless money pit, despite being tight with it)
Eventually some expensive people get hired to put the "finishing touches" on the product and they start the long slog to fix it up (or just quit in horror after a few weeks)
So far: 1) retired the entire project after years of development 2) company got bought out and the product scrapped after several years of investing money on it 3) is in progress (company has enough money to fund the situation for years, but time will tell if we manage to untangle the mess before it kills the project)
I imagine that after a month, when the senior developer(s) laid down the foundations, it should be possible to find work for the juniors (not all at the same time, but adding them gradually). They just cannot be assigned tickets at random.
1. Devs enjoy getting to use new languages in the real world, and helps keep us learning.
2. The better you know a language, the more tempting it is to take shortcuts. When you don't know a language well enough to write really gross things that work "for now", you have to think carefully about the simplest possible solution to a problem that you can express clearly in an unfamiliar language.
3. You learn tons the first time you solve a particular problem. At the end of the solution, when all of your hacks and tradeoffs are fresh in your mind, you are the best possible person to tackle all the shortcomings and immediately do a rewrite in a language you are expert in.
3. Management really can't twist your arm to just go ahead and transition the prototype to MVP. All you can really do is throw it out there as a public beta with no guarantees while you build the MVP properly using everything you just learned from doing it the first.
4. You can start collecting feedback on what users want included in v1 and get an idea of where the user base is heading with their desires and plan some of that into your design, again reducing long-term tech debt.
If your prototype sticks the landing well enough for management to decide to move forward to MVP, then it's good enough to mark your territory in the problem space while you do the rewrite. You won't lose competitive advantage while it's sitting out there in a separate VPC collecting users and activity.
In a healthy company, this isn't a difficult sell to management. They'll understand the short and long-term value to the company. In more toxic environments, it will look more like you're throwing a poison pill into your dev process—which you kind of are. So, you know, tread carefully. But when people buy in to this process, it works really nicely and has far better long-term outcomes.
These problems that take the most time in any project are mostly human, conceptual, product, and communication problems. Not code problems.
Once you have sorted out all of these problems and solved them, a rewrite in a different language with a better design moves very quickly. You don't have to double the runway or the time to MVP. Time estimations are always wrong anyway. But in my experience when I've gotten buy-in for this approach I would guess it adds 25-30% of actual time overhead to a roadmap. Prototypes always take longer than expected, and an immediate redesign/rewrite takes less time than expected.
Selling this approach to leadership really boils down to clearly identifying the value of developer time. It's not code; it's solving business problems. Once those are conceptually identified and solved, the design and code tend to fall into place without a ton of trouble. The problem is that no one really knows what the business problems are until you try to solve them and really dig into the details.
Prototypes should be understood as the process of defining the problem space and uncovering all the hidden issues that people haven't really thought through just yet. MVPs should be an actual product based on that exploration and problem definition and solving. What I'm suggesting here is really just a process boundary that reflects the difference between the two things.
"Can runway handle that?" is really a lot like asking if you can afford to build a product at all.
This has not been my experience. Much of the worst code I've run into was written by engineers that were new to a language or framework (including myself).
My larger point is that the first attempt to solve any problem in code is going to suck, so you might as well suck in a different language that won't ever have a chance of being deployed into prod. You learn some things, you get some exp with a new language, and a better frame of mind for solving the problem in a better way.
I have the opposite experience, the worst performance bottle necks I have had to trouble shoot are from principal engineers (and tech leads) that use medium to high level abstractions in a ruby on rails that don't actually understand what how many sql commands they are triggering by writing clever one liners that taco my db, from a rails 2.4 method that has a much better rails 5 system they don't know about.
I used to work with experienced and passionate Scala developers. On the one hand, their expertise helped them to avoid common language and runtime pitfalls.
On the other hand, the desire to produce elegant, pure functional code, experiment with and utilise powerful abstractions slowed the development down, led to the code that was hard to understand and jump on for new team members.
I saw many successfull companies with shitty software.
I had the impression the business part of things killed much more companies if they didn't get it right.
Day 1
Stakeholder: "The app is down, what is happening?"
Engineering lead: "We are working on it. Do not worry".
Day 2
Stakeholder: "The app is down, what is happening?"
Engineering lead: "Do not worry, we got this".
Fast forward to Day 15:
Stakeholder: "Can you explain why the app is down again?"
Engineering lead: "We got this"
Stakeholder: "That is what you said last 2 weeks ago. We will send someone to take a look at it"
External auditor: "wtf wtf wtf. These guys do not know what they're doing."
Day 16:
Stakeholder: "Say hello to Bob, he is your replacement. Please work with him making sure he has all the context he needs during this transition".
Engineering lead: "But, but... it's just technical debt"
Stakeholder: "You may leave now"1) Some people mistakenly interpret "maximizing the work not done" with maximizing tech debt.
2) Power is the ability to make people do what they do not want to do. Saying "no" is a way to exercise that ability. Some leaders become addicted to the feeling of exercising their power.
3) Cohesive groups of people tend to prioritize cohesion over rationality during decision making (i.e.: "groupthink"). Groupthink makes teams unable to make correct decisions or assess risks in a rational manner.
Many startups suffer of issues 1, 2, 3. And some even perceive those weaknesses as strengths. When that happens, the only way out is to decapitate the team, punish groupthink and reindoctrinate the team members.
[1] https://www.joelonsoftware.com/2000/04/06/things-you-should-...
> If you are writing code experimentally, you may want to rip up the function you wrote last week when you think of a better algorithm. That’s fine. You may want to refactor a class to make it easier to use. That’s fine, too. But throwing away the whole program is a dangerous folly, and if Netscape actually had some adult supervision with software industry experience, they might not have shot themselves in the foot so badly.
But I wouldn't go so far as to call it the sole reason the project failed - that just sounds too easy an explanation.
The from-scratch rewrite is a classic response to declining productivity. In the metaphor of technical debt, it's equivalent to declaring bankruptcy.
As Jamie Zawinski wrote in 1999: "We never distributed the source code to a working web browser, more importantly, to the web browser that people were actually using. We didn't release the source code to the most-previous-release of Netscape Navigator: instead, we released what we had at the time, which had a number of incomplete features, and lots and lots of bugs. [...] The code was just too complicated and crufty and hard to modify [...] By being a cleaner, newly-designed code base, so the theory went, it was going to be easier for people to understand and contribute. And this did get us more contributors. But it also constituted an almost-total rewrite of the browser, throwing us back six to ten months. Now we had to rewrite the entire user interface from scratch before anyone could even browse the web, or add a bookmark." -- https://www.projectseven.com/grafitti/october_2001/zawinski....
He doesn't use the "technical debt", because that didn't start becoming popular until 2004 or so: https://books.google.com/ngrams/graph?content=%22technical+d...
but if you look at Ward Cunningham's original 1992 description, it sounds pretty congruent: "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation [...]"
> There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
> It’s harder to read code than to write it.
> [...] Even fairly major architectural changes can be done without throwing away the code. On the Juno project we spent several months rearchitecting at one point: just moving things around, cleaning them up, creating base classes that made sense, and creating sharp interfaces between the modules. But we did it carefully, with our existing code base, and we didn’t introduce new bugs or throw away working code.
> A second reason programmers think that their code is a mess is that it is inefficient. The rendering code in Netscape was rumored to be slow. But this only affects a small part of the project, which you can optimize or even rewrite. You don’t have to rewrite the whole thing. When optimizing for speed, 1% of the work gets you 99% of the bang.
> Third, the code may be doggone ugly. One project I worked on actually had a data type called a FuckedString. Another project had started out using the convention of starting member variables with an underscore, but later switched to the more standard “m_”. So half the functions started with “_” and half with “m_”, which looked ugly. Frankly, this is the kind of thing you solve in five minutes with a macro in Emacs, not by starting from scratch.
Once the technical debt is so deep that bankruptcy seems like a good option, digging out is also not a great option. It's certainly possible now if it wasn't then; people like Michael Feathers have done great work charting paths out of an accumulation of garbage code. But at the time of JWZ's resignation letter, Martin Fowler's book Refactoring had been out for all of 3 months.
In particular, Netscape was faced with a choice: they could either do a total rewrite quickly, or take much longer to do a slow rewrite while still pushing forward, but at a much reduced development speed. That's fine if you're some large enterprise in a stable field with a solid revenue stream. But a startup competing in the then-fast-evolving world of the Internet? Competing against the monopolistic giant of Microsoft, who was desperate to dominate the space?
There's little reason to think a slow cleanup would have been any better than a fast rewrite for Netscape. Especially if their solution involved open-sourcing things, which a) requires a non-awful code base, and b) reduces the pace of refactoring and cleanup. So I think it's fair to say that either way Netscape was killed by technical debt.
Nothing has fundamentally changed between now and then. Sure, the information may be more readily available, and we can argue that it's unfair to judge Netscape by today's standards, but even so, I think the article you linked makes a strong argument that technical debt did not kill Netscape, and that their approach to managing it did. This is why it goes on to discuss other, more successful approaches.
> In particular, Netscape was faced with a choice: they could either do a total rewrite quickly, or take much longer to do a slow rewrite while still pushing forward, but at a much reduced development speed.
Were they actually faced with this choice? The article you linked points out there are at least 3 reasons people choose to do rewrites. This is one of them ("architectural problems"). Even still, the article's argument is that this is a false dichotomy, and that there are better alternatives.
I also think "total rewrite quickly" is an oxymoron and a bit disingenuous. Your article points out it took them three years. In my experience, they typically take full-blown products at least two years - if they are successful at all.
If you are worried about Microsoft beating you because you are slowed by technical debt, you certainly need to be worried about Microsoft beating you because you didn't ship anything for three years. That is the entire point the article makes. The very first sentence reads:
> Netscape 6.0 is finally going into its first public beta. There never was a version 5.0. The last major release, version 4.0, was released almost three years ago. Three years is an awfully long time in the Internet world. During this time, Netscape sat by, helplessly, as their market share plummeted.
I find your next point surprising, given your citation:
> There's little reason to think a slow cleanup would have been any better than a fast rewrite for Netscape.
The article you cite makes this exact argument:
> You are putting yourself in an extremely dangerous position where you will be shipping an old version of the code for several years, completely unable to make any strategic changes or react to new features that the market demands, because you don’t have shippable code. You might as well just close for business for the duration.
It then goes on to list alternatives, including a "slow" rewrite, during which time you can still ship code.
None of this is to say that too much technical debt can't slow you down to the point that you cannot keep up with your competitors - but it is certainly not the argument your citation makes. Its argument is tangential, and related only in the sense that technical debt is presumably what pushed Netscape to make the decision it did.
I have to say, I am fascinated by our difference in takeaways from this article.
A total rewrite, if done well, is going to take less resources and way less calendar time than a refactor-and-cleanup approach done equivalently well. That's what I mean by quick. Was Netscape's rewrite done well? I doubt it. But there's no reason to think their refactor-and-clean up would have been any better. Which again leads back to my point: they were probably fucked either way at that point. Note that in the ensuing years, nobody has made major direct revenue from selling a browser.
The thing that has fundamentally changed versus 20 years ago is major advances in dealing with existing code. Back then, writing tests was extraordinarily rare. Now if it's not the norm, at least it's common. Things like continuous integration are common. Builds and tests are much faster, and the amount of computing power available to an individual dev is hugely improved. The tooling advances are incredible as well, from debuggers to languages to runtimes to IDEs with built-in understanding and refactoring of the code. All of this makes incremental cleanups much easier that was possible in 1999 with a 1994-era mishmash of C and C++.
> A total rewrite, if done well, is going to take less resources and way less calendar time than a refactor-and-cleanup approach done equivalently well.
This is totally up for debate, and largely what Joel's article is talking about. Does "less resources" include opportunity cost? You may be able to get something out in less calendar time, but will it be as feature complete? Will it get rid of the old bugs that plagued the product? Will it avoid introducing new ones? There are many reasons to believe that a refactor would be better - for most projects this is probably true (at least in my opinion, and evidently in Joel's too!)
I, like many others, have seen the disastrous results rewriting from scratch can produce - if they produce results at all.
I'm not sure that the time builds take has any impact on which approach is faster. Certainly tooling advances might - but it's unclear to which approach the advantage is given. And while tests certainly make refactoring far easier, this can often be done as part of the refactoring.
And let's also remember that the article cited is written 20 years ago - the timeframe you're referring to. This idea that rewriting from scratch is dangerous and costly is not predicated on recent advancements in tech.
We can argue whether Netscape's approach to rewrite from scratch was the right approach to take, and we can also argue about whether rewriting from scratch is generally appropriate, but the intention of my original comment was to point out that Joel specifically makes the argument that it was rewriting from scratch that killed Netscape - not technical debt itself - and that there are alternatives which would've been far less risky (from his POV.)
It’s difficult to imagine Firefox having the success it had (being a browser for the modern web) if it was based on the Netscape 4.7 source code. I really think they had no other choice.
> Even fairly major architectural changes can be done without throwing away the code. On the Juno project we spent several months rearchitecting at one point: just moving things around, cleaning them up, creating base classes that made sense, and creating sharp interfaces between the modules. But we did it carefully, with our existing code base, and we didn’t introduce new bugs or throw away working code.
> A second reason programmers think that their code is a mess is that it is inefficient. The rendering code in Netscape was rumored to be slow. But this only affects a small part of the project, which you can optimize or even rewrite. You don’t have to rewrite the whole thing. When optimizing for speed, 1% of the work gets you 99% of the bang.
Quite frankly, Netscape was very good for its time. Firefox today is also drastically different than Firefox 1.0 ("It’s difficult to imagine Firefox having the success it had if it was based on the Firefox 1.0 source code").
It wasn’t even technical debt in the sense that it was bad code, it was because they prematurely tried to optimise everything, so adding a feature would take twice as long. We also couldn’t add new devs to the project because the team lead’s decision to not document the code, because “code should document itself”.
Bugs could only be identified by the dev who wrote that feature because the code was so obscure and undocumented.
I am looking for projects that are very small.
In projects with a lot of business logic, or projects that require understanding of a specific subject, it’s often better to write comments to clarify what’s going on. Sometimes code alone isn’t enough.
- Performance optimizations (always worth mentioning why you ugly'd up some code for a necessary speedup)
- Kludges where you're working around something in an external library or service
I've also had this experience, where a product accumulated so much tech debt (in part, due to my decisions) that it was simpler for the business to outsource the functionality. That wasn't the only reason, but it was a big one. When tech debt sand gets in the software delivery gears, the allure of vendors increases. Sure, the business won't get things exactly like they want them, but they won't have to deal with missed deadlines and degraded functionality either.
Well put. Could not have said this better myself. I've experienced situations where the tech debt sand is a consequence of fairly unique business and internal system constraints. The allure of vendors did increase and the company did eventually research vendors, but nothing addressed our snowflake use cases perfectly.
I don't think that's necessarily a positive reflection on our product though. If anything, the lesson I take from that is to build your business processes and teams to reduce as much accidental complexity as possible.
I can’t blame them for doing this tho, I often felt like we spend most of our time discussing implementation, doing code reviews and optimizing code for the sake of optimizing it. Towards the end it was also clear the business just didn't want to deal with our team anymore, they were annoyed we couldn’t just build the feature they wanted.
Code tells me what the coder was doing. It does not tell me why. If the why is not blindingly obvious, it deserves a code comment.
Basically 0. If you have a good idea - you can overcome. If you have a crappy idea and an amazing engineering team you are doomed.[1]
Technical debt, if it's relevant at all, is relevant at much later phase than that.
Too much debt slows you down, and if you don't move quick enough as a start up, the money stops. It's that simple.
If you’re reading this comment before you read the article: it’s about how to avoid getting caught too badly in the first place, not about magic bullets to get you out when you’re basically bankrupt.
That sucks. I have been there. Awful. For me, the silver lining is that I no longer ignore stinky red flags
Do NOT put up with that.
You don't have to. There are so many companies, even now, where you can get decent work with much more sensible norms.
There's always kind of a gravitational pull toward excess debt because "get the [fix/feature] out there in front of [users/client/exec]" is such a tempting drug for management, even if they're generally good. But the amount and toxicity of the debt varies widely. Find a company that's sane about this if you value your own sanity imo.
My experience is a bit more on the agency side where technical debt is caused by developers/project managers not really caring about the longevity of an application. The idea of going back and fixing anything that isn't specifically called out and invoiced for is so far away from the priorities of these businesses.
In the corporate world, on the other hand, the problem is an astonishing fear of originality. Most places I have worked have invented here mentality baked into every decision. Every 5 to 8 years they start over completely with a different tool, framework, or language hoping it magically solves for the group’s prior poor implementation.
This resonates with me. I consider myself a better fit for product focus vs project focus, preferring to find something meaningful and nurture it in terms of value to customer, and value to developers (code quality).
Recently I accepted a job at an Agency, which a coworker described as a "chop shop". I became depressed. The app I would work on, got frenzy activity from Oct - March. It had to be released in March no matter what. Between that time there was no activity, no refactoring, no addressing tech debt. In fact, one week in December the client was unable to pay for some technical budgeting reason and we all just sat around for that week despite looming deadline.
Well, over the years this thing accumulated tech debt. Even though it was a cash cow for the agency: reliable income at reliable times from a good customer. One would think the health of the product would somehow invite extra effort. Every year new devs would start working on it (new devs, because turnover at these places is high). They would look at the mess and begin resume driven development to use the shiniest new things that were left out, hoping to fix and compensate for the problems from just sheer neglect, and then move on for the cycle to repeat.
That said, there are people that just prefer hopping from project to project to experience new things.
But, if you have repeated functions, or two similar react components which could be merged, that's a far lower risk to current and future development velocity.
But when they play a piece for the first time the performance is typically much lacking. They have to rehearse the symphony or rock-opera or whatever.
Removing technical debt is like playing the song again, this time better. You learn what works what not and can do it better. The challenge is how to know the code-performance is good enough.
Personally I've never seen a startup that was actually killed by tech debt, although it does sometimes cause good employees to quit, and of course it causes product delays. It's like any form of debt: a dangerous but oftentimes necessary tool.
If there is a bug in the woods, and the bug takes a shit, but no one is around to hear it...
Now, is it going to kill a startup? Hardly. At that point, an MVP, it's probably still small enough that the better decision is to rewrite it altogether. If you split the server and the UI to begin with, you can re-design the front-end and make a party out of it, try to make it pretty this time.
If you're at the point where you need more than 1 person to handle the whole app, then you're not a startup anymore, imo. You have a proven small business.
1. Pledge to schedule 20% (pick your percentage) of the sprint as tech debt payback rather than stories. The problem here is that this stuff tends to drift to the bottom of the sprint and then slip if anything else is delayed. This is where most teams start as you can start at an arbitrarily small percentage and scale up gradually.
2. Fixit Friday; the middle Friday of a sprint is for fixing tech debt, not working on sprint tasks. Engineers are encouraged to advocate for tasks they think would speed us up, and we also have done themed pushes where e.g. the whole team chips in to migrate the codebase to a new linter. One issue here is that it’s hard to make progress on larger initiatives, which leads to:
3. Fixit Sprint. At the beginning of the quarter there is often a bit of downtime as the leadership digests the last quarter and plans the new one. This can be a good time to take a sprint off product work, and really dig into some substantial re-tooling or refactoring.
Depending on how much resource you’re allocating to tech debt, you can mix and match these, I’d say 1 & 2 are probably mutually exclusive, but you could combine either with 3, or just do 3 instead.
I think it’s important to be clear about when you are taking on tech debt, and when you have cleared the hurdle and are paying it back, and clear rules like these help to communicate that message.
The company either thinks strategically or it doesn't. If it doesn't then it will only hurt you if you do. Just deal with it, work in the way they except (i.e. create even more tech debt for some fast results) and once it all crashes, tell them the whole thing needs a full rewrite (which you know will fail anyways). Not really much you can do here except for leaving.
If the company thinks strategically, then no need for something like a 20% schedule. You should be empowered to make these decisions yourself and you should sometimes create a lot of strategical tech debt and sometimes take a month to fix something if you know it will pay off. You are the expert - you need to understand the business for these decisions and then you communicate back (as you already mentioned), but that's about it.
Very few tech leads or devs are empowered enough to tell upper management they aren't going to work on the tasks assigned to them and will instead work on fixing a technical debt issue. And I can relate to this, because if I asked a dev on my team to work on building out a new report a user asked for and they respond they're going to fix technical debt instead, I wouldn't be happy.
Also we already know how this ad hoc approach to technical debt works in most organizations. You get 0% to work on technical debt.
But everyone is already doing that _to some degree_. Everytime you do a mini refactoring, and be it just some code formatting, isn't strictly "working on the tasks assigned". Unless you define "working on the tasks assigned" a bit wider, which you should.
> And I can relate to this, because if I asked a dev on my team to work on building out a new report a user asked for and they respond they're going to fix technical debt instead, I wouldn't be happy.
But you, as the lead developer (or however you call it), should have some insight on the longer roadmap. And if you see that there will be a lot of new reports necessary in the future and the current reporting-system has a lot of tech debt, you should decide if this tech debt should be tackled and then align with your team to do it. And this is what my point is: you should not even have to tell your manager (unless he is the one to decide if the tech debt is tackled or not).
You should have all the necessary information and decision power regarding resources, priorities and technological&business strategy to make this decision. If you don't have that, your manager needs to have that _and_ needs to have the competence to make the decision about refactoring without having to rely on your or other technical people. If he can't do that (which I've seen is often the case) then this is where the problem lies: you are not empowered enough and you can't make the decision yourself, but you should be the one and not your manager.
In the same sense, you should not be the one to decide how your developer implements the new report. Maybe he will decide to pull in a new library, maybe not. Maybe not doing it would create some new tech debt or the other way around. But that should be his concern, not yours. Your job is to tell him what the priorities are and that's about it.
> Also we already know how this ad hoc approach to technical debt works in most organizations. You get 0% to work on technical debt.
If the decision makers don't care and you can't educate them about that you should be empowered, then I believe there is no point in fighting. It will just make everyone unhappy. Leave or create a lot of technical debt while moving fast until the company is stuck. Who knows, maybe that's better? The market will make the decision.
I think by talking explicitly about tech debt you can get the non-technical stakeholders on board. (It helps I think of you were warning them about tech debt when you were taking it on, too, and not just starting to educate when it comes time to pay back.)
In my experience teams run better when everyone is somewhat aligned on how much tech debt to be taking on or paying back.
I am sure there are other ways to get alignment too; I’m interested to hear what other leads have implemented. I could believe for some team compositions the “let each engineer figure it out” approach would work well, but I think that is kind of like the Agile “self organizing team” — it’s rare that you get to work in a team where everyone is experienced enough for a free form approach to work, especially when you are growing fast.
Your CEO (or whatever stakeholder you mean) should not care about _how_ you work, they should trust you to do the right thing and enable you to do so. They should also give you feedback and correct your course, but only in terms of what you work on and with what priority. They should not care how you do it.
In the same way your lawyer does not tell you what kind of legal internal documentation they do and if they go to court X or Y to ask for something, you should not tell that to your stakeholders. You tell them only high level things that are relevant. Tech debt is not one of them, progress and estimated dates is one. And just like how the lawyer recommends you a certain path, you should recommend one to your stakeholder. E.g.: "Yes we can build this feature, but we could also build this other feature, which is almost as good and takes half the time. I suggest to do it this way".
> I think by talking explicitly about tech debt you can get the non-technical stakeholders on board. (It helps I think of you were warning them about tech debt when you were taking it on, too, and not just starting to educate when it comes time to pay back.)
Let's say you are in the position to do so and they trust you. Do you think it is productive to "warn" them? Let's be honest: even for developers it is very hard to understand the direct and indirect impacts of tech debt. And that is even though they can directly experience the effects. How difficult do you think is it for people for whom this is just an abstract term?
TLDR: I think it is easier for you to understand the business goals and make a technical decision than it is to make business people understand the technical constraints and educate them well enough to take the right decision.
I like your second sentence though! If explaining technical debt helps with the "us" culture, it should be done.
It's just that, from my experience, it usually does the opposite. And I mentioned the reasons: it is too difficult to explain and to understand, even for developers. Also, the most important aspects/impacts will almost never be measured - not talking about development speed here (which is already too difficult to measure), but motivation, attrition, broken window behavior and so on.
A better way to create an "us" culture as a developer is to make sure to be involved into product decisions from the very beginning, including contributing ideas. And being hold to the except same KPIs/OKRs as other departments. That also means, as a developer you need to have the discipline to heavily sacrifice code quality and create technical debt sometimes and you might have to push this against other developers. That is the price to pay.
I fundamentally agree with this approach and there are companies that do as well. What is the term that encapsulates this sort of culture? Is there one?
If there is one, is there a reliable / repeatable way to determine during the interview phases if the company you're interviewing with is this type of culture?
I would say this is one of the strengths of the culture at my company and we spend a lot of time trying to get that across to engineering candidates. I'd love to improve our messaging there if we can, though.
We talk about a culture of trust, that we hire good people and then trust them to do their jobs. We talk about avoiding hierarchy and beurocracy where it doesn't help us. We talk about engineers driving their own work forwards and we make it clear that we're looking for people who can do that.
I wasted lots of time prematurely paying back tech debt only to have that code later deleted or rewritten.
It is important to properly prioritize tech debt and pay back things that the business is currently paying high interest on.
Once the product matures it makes sense to be more proactive in doing things right, but in the early stages, things are changing too fast that you just need to get out a rough prototype.
Also, if you're growing fast enough that it matters, you'll need to redesign the whole thing anyway. Google did that on their search engine at least five times. In the early days, it took weeks to update the index.
/s obviously but this is more real than it should be
Once you know that your company is there to stay, you hire experienced people and create a strategy to pay it back.