Ask HN: How often does technical debt lead to the failure of a startup? | Hacker News Reader

Ask HN: How often does technical debt lead to the failure of a startup?

40 pointsYadi54 comments

The question here comes from my concern of how some startups can actually survive with high technical debt with customers? Wouldn't that lead to their failure as a product/service sooner or later?

I work as a web architect contracting with startups on smaller complicated pieces of technologies. These days lots of startups I talk with have a very high technical debt in their product(s) and most of them don't seem to care.

My first advice to them is that things like that can kill a product, but I've had some strange feedback after passing on that advice.

So I wonder if technical debt does actually have pitfalls along the way? (Bootstrapping yeah I agree, but not when you have 8 engineers on the team)

View on HN

The likely failure of a start-up is market based, not technology. Look at Twitter's fail-whale years. They were on a platform which couldn't scale to their growth and it took many months for them to re-architect and build a working solution. It definitely didn't kill them.

Another example is Instagram, I read recently (but can't find the link) that the founders were learning to code when they built it, and that some of that code is still in production, and apparently, it's embarassingly bad, but it gets the job done.

On the other hand, I worked for a consulting firm and one of our clients sites was absolutely riddled with technical debt. The result was that any changes they wanted to make were significantly more expensive than if they would have been on a better architected system. The thing is, it did't matter to them because the cost of re-creating the system was significantly more than the cost of working with what they had.

I hate technical debt, but have come to realise that it is a debt much like any other, and if you can cover the cost of that debt, it doesn't really matter down the line.

This isn't an excuse for bad programming, I consider that to be something slightly different from technical debt which often works but is not an idea implementation.

Thanks, I agree there are a lot of factors for a startup to fail.

Thought in the context that I mentioned, it is about some startups that have surpassed the point to find their first thousands of customer and now they need reliability.

But again there are different interesting tradeoffs of these issues.

Regarding:

> Another example is Instagram, I read recently (but can't find the link) that the founders were learning to code when they built it, and that some of that code is still in production, and apparently, it's embarassingly bad, but it gets the job done.

I was at the OpenAir conference this year where Mike Krieger mentioned the above story. But I'm not sure if the Video is out.

It is more important to sell software than it is to write it.

I don't know a real number but my guess and personal feeling, having failed and succeeded, is that technical debt accounts for very few failures. Wrong product fit, ignoring customers, failing to market properly, failing to ship etc all are far more likely to cause the demise of a startup.

Also, I would say to much concern over technical debt can cause failure. I personally lost what I conservatively would say was 30% of our net revenue last year letting my prior lead dev address technical debt that never benefitted a single client and that in areas we have now replaced (or are in process of replacing) entirely in the product. I am as much (if not more so) to blame as he was because I approved the time.

Indeed, there are lots of factors to lead to a product fail.

Though, I'm really curious, mind if I ask what sorts of technical debt you had in your startup that led to such massive slow-down in the process?

Were the debts among any of these:

* Documentations for on boarding devs. * Lack of tests. * Delayed refactoring. * Lack of alignment to standards.

So in my case, I will share that we screwed up last year by addressing things that didn't benefit our clients. We had probably 20 features that should have been developed that would have benefitted our clients. Instead I allowed our team to focus on scalability (which wasn't really an issue at the time), plus we migrated databases, added ElasticSearch for full text search (which wasn't an issue yet), reengineered a number of features, all of which were working, although maybe not ideally.

Also, almost everything you mentioned is on my list of WTF was I thinking. We wasted time adjusting code to meet new standards, adding tests to things we knew we were changing, adding scalability to places we had no scalability issues with (yet), adding new interfaces/code in places where something "might" happen etc. None of these changes benefitted a client directly, so IMO we lost money on every one of those initiatives. The reality is a lot of the code we "brought up to standards" was rewritten anyway when we started adding features and refactoring the old code to interface these features. I also allowed the team to create their own modules instead of extending some existing node.js modules that existed in the community, which to me was a waste when I saw what we did. We would have done better to extend the community modules and open source them back to the community.

As for documentation, honestly, we have pretty good documentation, although I wouldn't say it is without issues.

Overall I considered last year, the year of the "it might happen" screw ups. What really pissed me off is I spend a huge amount of time advising clients not to touch old code unless they know it is causing them a problem now or can be proven to cause an issue when testing against a new requirement. Yet, I still failed to follow my own rules, shit happens.

Thanks a bunch! You're a champ.

Yeah sometimes making these sorts of decisions could easily lead to big pitfalls on the way.

Regarding this part:

> "I also allowed the team to create their own modules instead of extending some existing node.js modules that existed in the community, which to me was a waste when I saw what we did."

This is technical debt as well by the definition of Wikipedia and some 101 CS books. Having enough contribution from a community could always be better of being used, unless that part of the technology is propriety in the product.

You are welcome.

I totally agree I screwed the pooch and having 20 years of experience didn't save me. Ironically I had made sure we were giving back to the community but missed some opportunities that would have saved us time and money. Oh well, this year we will add the code to the modules we use. It is on our to do so we remove unnecessary dependencies we added.

What if the company is in a tight market and it differentiates itself by being reliable and be able to constantly implement new features as well as competitors features quickly? I would assume taking on technical debt, unless you can bankrupt your competitors in one blow, is suicide since in the long run whoever have the least technical interest payments wins. That assumes the company has done those four things you mentioned solidly.

I feel you are talking about a situation that rarely exists. It does happen at least somewhat when you see a large incumbent versus a startup, but in those cases a startup is generally nimble and has a fighting chance.

IMO, if technical debt is what destroys your startup then you probably so poorly understood the customer problem early on that you engineered a solution that was going to fail either way. In all the companies I help I almost always see bad technical decisions, but their marketing is what is hurting them the most not their tech debt.

I see a lot of answers about how it's not technical debt that kills startups, it's failure to address customer needs. But I don't think these two are so different.

My definition of technical debt would be "problems in the code base that make it harder than it should be to do work," whether that work is adding new features, fixing bugs, scaling up, etc. The startup that can implement new features quickly because their code base isn't a disaster is more likely to succeed. They can try out new features quickly, seeing what customers respond to.

Customers need things like uptime, minimal bugginess, quick bug fixes, etc. Technical debt reduces uptime, increases bugginess, and makes it harder to fix bugs.

I think it would be hard to point at a startup and say "technical debt killed it". In most cases I imagine there are many inter-related causes of failure, but an inability to quickly add new features, fix bugs, and keep the system stable can't be a good thing!

Customers need things like uptime, minimal bugginess, quick bug fixes, etc.

You've basically elevated these items to the top of the list, above all other attributes, and declared them more important to all customers.

Customers will pay/use a product that delivers value. The items you listed subtract from that value but they may or may not eliminate it. It depends on the product, and how valuable those items are to the customers.

Edit: I'll add an example. Let's say you like video games. There's a particular game that you love to play. Do you love that game because "it's stable, has minimal bugginess, and the developers release patches on time"? Probably not. You love it because of the gameplay/storyline/graphics/whatever. If the game is unstable, that's irritating, and makes it less enjoyable.. but that doesn't change what you love (value) about the game.

> You've basically elevated these items to the top of the list, above all other attributes, and declared them more important to all customers.

No I didn't. I'm just saying that these things are also important to customers. Churning out tons of broken features on top of a constantly crashing/unavailable system is not a winning formula for success in many markets.

It might work to some degree for free consumer-facing apps (like Twitter), but for many other sectors reliability and data integrity are very important factors (like any B2B product).

Customers need things like ...

Definition of need:

a situation in which someone or something must do or have something

Saying "these things are also important" and that it depends on the type (twitter vs b2b) of product is a lesser statement than your original statement. If you want to tone down what you originally said that's fine.

Thanks, yes it's hard to point out one specific example of a startup, because it can be an inter-related cause as you mentioned.

Though someone else here mentioned how friendster lost their game due to technical debt:

I looked it up and it looks like that indeed it was an issue.

http://highscalability.com/friendster-lost-lead-because-fail...

> "Technical debt reduces uptime, increases bugginess, and makes it harder to fix bugs."

Yes, that is one of my concerns usually. It's good to go fast and build more, but not to an extend where it slows the process down on longer run.

Thanks for that link.

The comments are interesting, but the closing one by a Mark Peterson again goes to the bottom line of unaddressed technical debt by putting a date on the best known general technical problem:

Friendster failed in the summer of 2003 when it never took less than a minute to log into the site. Where it never took less than a minute to visit any page. Even pages that should/could have been static.

Debate about any missing feature after summer 2003 is less than moot.

Especially since added features or allowing viewing by the not logged in would tend to further slow down the site.

I have worked at a dozen startups, some failed, some grew and still exist, some were acquired and people made a shit load of money.

I have never been at a startup that failed for technical/technology reasons. All failures I've seen were due to issue with delivering what customers were interested in paying money for. Most often, product owners and developers were not able to establish good communication, which led to developers basically creating a product that didn't match what customers wanted. Period.

I would be interested in your definition of technical debt though. What some people call "technical debt" or "hack", I often call "the best/fastest way to address the current business concern and deliver immediate value". In a startup environment I think these is a critical workflow.

>> " Most often, product owners and developers were not able to establish good communication, which led to developers basically creating a product that didn't match what customers wanted. Period."

Thanks a bunch for pointing this out! It's important for me to hear this here :).

Regarding my definition of technical debt is this:

* Lack for proper documentation where on boarding devs is becoming a burden.

* Lack of written tests to a point writing new features is almost inevitable from breaking the product.

* Delayed refactoring which has led to serious performance issues.

* Lack of alignment to standards where the engineers have written everything in Javascript like a 5 years old painting all the wall with same color, rather than using a different language for some specific purposes.

Some other Technical Debts that I meant were:

* How engineers go on a trajectory of writing code where no one knows something new have been added.

* Lack of clarity who have done what, regardless Code commits, this one is about knowing who lead a milestone or a feature to production.

At one startup I worked at, most of the developers were very young and had their first job, nevermind first job with stock options as part of compensation.

When it came time for a round of financing, I was leading a team meeting when the subject of dilution came up: "Are we being screwed?" And I had to explain that a smaller percentage of a company with a higher valuation isn't "being screwed." and attendant to that we discussed cost of capital, classes of shares, preferences, ratchets, etc. You could see the light bulbs coming on as I explained.

Engineers are smart. If you explain to them that "good enough with the money we've got" will make them more money later, they are usually OK with deferring both money and correctness gratification.

Think of it as building empathy for "product owners" who might otherwise be dismissed as beancounters.

> If you explain to them that "good enough with the money we've got" will make them more money later, they are usually OK

That's true. Some engineers won't take the "just make it good enough" approach, but I believe most will - otherwise they don't belong in a startup.

I wasn't mentioning the broken communication between PO and devs in this regard though. I have seen teams where the PO were not able to describe what they wanted, what business value they wanted to add to the product, devs had to come up with their own description of feature and sure enough the market didn't care of that.

PO is such a critical position, and such a difficult one to fill. I can count on one hand the companies where I worked who understood that. A PO is not just moving tickets around in JIRA, they are in charge of describing in actionable terms the business value the team is trying to produce.

Agneed: Weak PO or weak product management will sink you.

@Zigurd, this is some Dev- fron-tline deep wisdom right here!

> 'Think of it as building empathy for "product owners" who might otherwise be dismissed as beancounters.'

Technical debt isn't usually directly responsible for the failure of a startup. However, it can lead to problems that can cause a startup to fail. For example, technical debt can cause products to ship late, new features being harder to build out without refactoring, poor test/QA systems in place, new engineers taking longer to become familiar with the codebase, etc.

Anything that delays shipping is going to contribute to the failure of a startup.

Thanks, that is indeed the on spot point I'm looking for.

Drawing some conclusions to what technical debt could eventually lead to at the end.

> "Anything that delays shipping is going to contribute to the failure of a startup."

Edit: forgot a word.

What does a startup absolutely have to do? It isn't creating the most elegantly designed piece of software in the world. It's creating value for the customer.

Along the way to delivering that value, sometimes a less than perfect solution is chosen (often because of resources--time, money, or lack of human resources) OR the perfect solution is chosen, but the product changes and that solution no longer is perfect.

Technical debt should be managed, but it can't be eliminated... unless you want to give up producing anything new, and just work on polishing your previous work forever.

When will it kill your startup? If it turns out that your product is NOT what your customer actually wants, but the shortcuts (tech debt) you took prevent you from making the necessary adjustments to meet your customers needs.

Thanks. Indeed, I agree with you regarding how startups are about "creating value for the customer."

But I think at some point, this might become very difficult due to the technical debt when the startup can't deliver more value than it promised.

That's definitely true of technical debt.. There are two issues here:

1) A competitive issue.. a company will less tech debt will be able to improve their product faster. That is a risk, but it depends on the competitive landscape how much risk that amounts to.

2) When you say "startup can't deliver more value than it promised".. this is more a variation on my original point. The issue isn't "delivering more value", but that the value that has been delivered is insufficient to meet customer needs.

Above that, "delivering more value" is a growth issue (your ability to meet the needs of additional segments of the market). And it's certainly true that startups with less tech debt can improve their product faster -- ie: grow faster.

Thanks :)! Lots of good clarified points.

I don't have a story on startup failing due to tech debt, but I do have a story about technical debt forcing us to pass on potentially acquiring a startup.

A potential acquisition came our way that seemed to be exactly what we were looking for. Customer traction and revenues, the features exactly what we were looking to overhaul. There was very high internal interest in acquiring the technology and the company.

The need for this set of features was so high and the internal development estimated to be so costly (in developer time and delays in other projects) we were willing to overlook that this app was not built on our primary stack and only a few of our developers were familiar with. The language choice made by the company didn’t cool our appetite. However, technological choices made us pass on the acquisition.

They didn’t use a framework.

This means that our developers would have a very long learning curve. We also saw a lot of code that did what a framework would have taken care off and this means that we would have had to learn, maintain and expand the that code instead of working on the revenue generating features.

We found close coupling of code all over the place. This meant we couldn’t quickly extend and modify the features as we wanted without first paying back the technical debt accumulated by the developers.

It wouldn’t have mattered which framework they would have chosen as most of them have good documentation and force some sort of standard development practices. However, we couldn’t take a chance that all the behind the scenes stuff would need a rewrite if we wanted to expand and scale the platform. We passed.

Heh. I once joined an early stage startup that had a big bowl of spaghetti code, no source control and no framework. My first insistence was source control. Then after a year in which we hired several developers, I insisted we move to a framework so the learning curve wouldn't be so steep for new people.

It was a monumental effort that took more than a year to complete, and we did the "brain transplant" (as we called it) at the same time we were moving to a new data center. Launch was shaky but overall it went very well. One of my fondest memories is of the "war room" on launch day as we triaged incoming bug reports. The team worked great together and at the end of the day we felt like we'd really accomplish something.

Sometimes of stress can be very rewarding.

Yeah, that reminds me of one of my roles in a company when we were deciding if we wanted to ally with another one. I'd be tasked with spending some time looking at their code base and talking to their technical people to decide if the alliance was technically worth it. Can't remember any of the judgments I made (was a quarter century ago), but my input was an essential part of the decision.

This is an awesome reply! Just awesome.

Folks rarely notice how much a decision making matters on this regard and technical knowledge matters a lot on the way. Since having a company* acquired sometimes could be a good business return, but in the first place due to hitting the pedal it has brought lots of technical debt which makes an acquisition impossible.

Can you elaborate on the scale involved here?

Was the acquisition price in the 100k, 1M, 10M, 100M range?

Was the cost to replace/rewrite the systems in developer-years estimated in the single-digits, 10s, or 100s?

I have seen it happen. But I have also seen startups skate by. Cost of capital is a real issue, and spending money on being perfectionist is not worth it if you arrived at a good-enough implementation despite being naive and cheapass.

More often it's not "technical debt" but outright failed implementation when technology kills a startup

See how far it takes you and if you are wise enough not to press your luck, you'll have the chance to do it right with cheaper money.

See this is what I suggest most of the time!

My first startup that I built and sold had 0 test written, but I knew on spot what I'm giving up for what. I always say that last line of yours:

Making "the chance to do it right with cheaper money." rather than a whole team to re-do everything from scratch or break down already running product.

Sort of goes back to your point with cost of capital, indeed.

Thanks, good points :)!

There is no "TASB" for appraising Technical Debt like there is an "FASB" for Financial. So most measures resemble the parable of the Six Blind Men and the Elephant.

Trade off an extra day to do each release versus 30 days to fix the release process? Trade off an extra week to add a feature versus a month to refactor and a month of customer turmoil on the refactored release? Trade off a month of getting complete test coverage versus saving two days on a future feature development?

My guess would be that the startup founders are doing a better investment calculation than it might appear to you.

Indeed, founders usually have a better complete vision the the folks around them.

Though I was just pointing that out for over time. One time around the tech-debt is not that big of a deal, until it becomes a beast and hard to overcome.

Try framing it as interest payments:

When you get X customers, you'll have to commit Y resources to fix this one thing.

Taking on technical debt is a smart thing in early stages because it's essentially low-cost, non-dilutive financing. So you should recognize that very real benefit.

However, if they are young startups they might not even realize they are doing this smart thing. You can add value by telling them are smart and giving them the schedule of technical interest payments they will have to pay one way or the other if they are still in business.

Hey that is another interesting point, thanks:

>"the schedule of technical interest payments they will have to pay one way or the other if they are still in business."

Glad you found it useful.

Another reason technical debt is useful, as touched on elsewhere, is that it's something of a real option.

Options have value - think e.g. Oil futures which are the right but not the obligation to purchase X at Y price in the future.

Real options also have value - certain investment decisions can be delayed (e.g. building a plant or refactoring code) and by delaying them you create an option. Here the option is the right but not obligation to write better code that will actually work :)

I know it's hard to convince clients to change their ways, I have found that telling them they are doing great things they didn't even realize is a good way to start before you "bring the pain". Best of luck!

In both the best and worst cases, technical debt gets 100% relieved. In the best case because growth requires an entirely different approach to solving rather different problems, and in the worst case because the company goes bust. Where technical debt matters is in the middle, lifestyle businesses and Enterprise IT and companies that aren't going to come close to monopoly due to market position.

Good luck.

Not all problems really count as technical debt, IMO.

To figure out which ones do, I sometimes think about the correctness vs. completeness spectrum.

Correctness is how well the architecture fits the problem, how well edge cases are handled, if it's a service then whether it's built to scale horizontally, etc.

Completeness is how shippable the thing is. Are the important features there, are the bugs not too severe, could the thing live in production without causing severe harm, etc.

Some problem domains live only at the far end of the correctness spectrum, like software that life depends on, flight control, nuclear reactors, etc.

Other domains might be all the way on the completeness side, like the throw-away prototypes, temporary stand-alone marketing sites, etc. If it works at all, you can ship it. Problems there don't count as technical debt.

Obviously most projects are somewhere in the middle.

Most high-growth companies can get away with erring a bit toward the completeness side, and ignoring correctness for many things, because a lot of stuff will be re-written every year or so to handle the changes in the business brought on by customer growth. Since it will be re-written anyway, it might not be useful to think about the lack of correctness as technical debt because you won't ever pay the debt. The real trick is figuring out which things must be perfect now and which can be improved later.

Getting that wrong either way could lead to the failure of your startup. Settle for nothing but perfectly correct architecture for everything, and it might take too long to ship. On the other hand, ignoring severe problems in a critical system might somehow limit growth, cause a legal problem, etc., and eventually sink the company.

Thanks, that is a very good metric that you shared with an example:

> "correctness vs. completeness spectrum."

and yes! It's a double edge sword:

>"Getting that wrong either way could lead to the failure of your startup."

It has been asserted that a reason for Friendster's demise was in part internal technical problems that manifested themselves in high page response times, often to the point of being unusable.

Though, much of this was exacerbated my management's insistence on implementing a myriad of side jobs to bring revenue while deliberately ignoring the main part of the website being unresponsive.

That's interesting. Thanks for pointing it out, after you mentioned that, I had to look it up:

http://highscalability.com/friendster-lost-lead-because-fail...

> "technical difficulties proved too pedestrian for a board of this pedigree. The performance problems would come up, but the board devoted most of its time to talking about potential competitors and new features, such as the possibility of adding Internet phone services, or so-called voice over Internet protocol, or VoIP, to the site."

My previous employer failed after almost ten years of venture funding (mostly because it is pretty hard to make a good return on ten years of investment). We had a lot of technical debt. It did not kill us, per se, although I think it was correlated with problems in the company: a lot of things would have been easily resolved had we been less pressed for engineering time. We never had enough revenue to grow the engineering team significantly, and what engineers were there were often fire-fighting. There may have been indirect causal effects like causing good people to get frustrated and leave, or simply causing good people to get frustrated and avoid working on certain problems. We had a few people who flipped out and started repaying the principal (including myself), but they were taking limited engineering time from things that the company needed day-to-day.

It may be true that technical debt directly kills some products, but if it's bad enough that you can't get certain things done, even if what you do instead is also important to the business, the debt is a sign of bad engineering management (either in the past or the present). And bad engineering management will kill the product.

Technical debt is more manageable when the core business data is accesible and in sort of usable state. Otherwise a lot of time is lost on improving the state of the data. Time that could be used to create features people will pay for. That's the only constant I've ever recognized in failing startups

Thanks. Right, perhaps having some pieces of a technology solidified could help with moving on to new features.

For example an API that users pay for would only be updated or relied on when it's debt free.

Charge money when something is debt free? I'm interested in learning more about that, please.

Never. It's almost always failure to find customers or running out of money.

I'm still unsure exactly what technical debt is?

I haven't looked for a precise definition, but it's what you will have to spend in the future for problems that exist in your technology today. Those problems could be outright bugs, or design choices that will make it more costly to continue development work in the future as the product and business follow a normal course of growth.

It's a metaphor:

https://youtu.be/pqeJFYwnkjE

Essentially how well you can write a program to reflect your understanding of a problem. Though it has come to more generally mean writing crap in a lot of cases.

It's a credit, just like in banking but instead of money you're borrowing time.

Reply on news.ycombinator.com