The “bugs are 100x more expensive to fix in production” study might not exist

https://news.ycombinator.com/item?id=27892615

It's just an idea that you can use in an argument. When you invoke this idea, what you are actually saying is: "I want to prioritize this bug now, because my experience tells me it will save cost and/or effort." You use this rhetorical device to put weight behind your argument. It's like invoking "80/20 principle" or "technical debt".

Maybe I've become cynical but the longer I've been working, the more I like the postmodern theory that almost everything is a language game. I start to recognize bullshit, and to recognize when it's the right time to use BS myself. (Not saying that truth doesn't matter. Just that sometimes you have a correct but not convincing point, and you can support it with a not-100%-scientifically correct, but more convincing point.)

That will be this much costly even if you try to find those in pre-production.

Corollary: The most expensive bugs to fix are the oldest bugs.

For some other bugs there might not be a difference. There cannot be a general answer to this question, there are too many factors involved.

Modern development makes some things easier than it was in the past. Then again there will also be new problems.

The neat 100x value, the lack of context, it sounds more a saying from grandma, which is wise, but not to take at face value.

Sometimes (rarely) it is actually cheaper to fix in production, sometimes, it is way more than 100x. I had a coworker who worked with IC cards and fixing a bug in production meant scrapping a whole batch of cards (can be millions).

The right question is not wether it's more expensive to fix bugs in production or earlier. The key question is:

Who bears the brunt of the cost of fixing bugs at one stage and another?

This is a parallell to Conway's law: Which cost-center benefits for NOT working with agility (not rituals) and using Dev(Sec)Ops (collaboration) to improve processes and techniques?

It's that simple. Anything else will not get oxygen to thrive, simply because of no allocated priority and funds to bridge the disconnect.

What to call this law? Maybe "The Law of Cost Centers", or something similar.

Anyone trying to fix this flaw will be fighting windmills head-on every step of the way. The big international corporations will have budgeted this due to scale and a must-have need, to be much more available and reliable than smaller shops.

1. You measure the average cost of fixing bugs before they go to production. You also measure the average cost of fixing bugs after they go to production.

I think this measurement is uninteresting, this might be because easy bugs get fixed first, and they are naturally low cost. Nothing to do with being in production or not.

2. Identify 2N bugs before going into production. Randomly select N bugs and don't go to production before they are fixed. Then go to production and fix the other N bugs after a while.

I think this would be more interesting, but I suspect that nobody does this, as this measurement itself is possibly more costly than all of the 2N bugs together.

But you don't have to go to that extreme. For instance, I just heard today of a lost contract from a coworker: the client was fed up with a call management system that was dropping calls like crazy and nobody was able to fix since you could not even replicate the system (the last time they tried it, the test system pulled stuff from the production db and it basically brought everything down).

How do you calculate the cost of this? Cost of debugging, lost revenue, lost referrals, brand damage, and the toll on developers and everyone doing crisis management.. I'd say 100x is a very optimistic estimate. It can very well cost much more than that.

Trivial bugs are trivial bugs, and not all the bugs are the same. A misaligned table is nothing compared to a monolith/blob of 50Mb (either a binary compiled from C or minified JavaScript) that crashes with "segfault" or "undefined" but runs fine in debug mode.

Bottom line: bugs in production, especially on complex systems, can be very hard to identify and fix - and sometimes can cost you your business.

There are projects where nobody cares much about bugs in production and it would cost a lot to make sure no bugs enter production. These projects typically tend to rely on feedback from users.

Then there are projects where production errors are just not acceptable (think Solid State Booster controller or ABS break controller). These projects tend to fail catastrophically in case of a production error.

Most projects are on the spectrum but that spectrum is very, very wide.

Because I like to produce correct code and I don't like to be pressured by the "business" to hurry up I tend to choose projects where errors tend to be costly if they get into production, but other people may have different preferences.

Now, it is important to understand what is the cost of the bug.

We kinda understand cost of production bugs (the damage to company + all actions necessary to remove the bug and any lasting effects).

The cost of removing the bug before it gets into production is less well understood. If you invest in better developers or put automation or work to imbue your developers with "drive for excellence" or delay it to give them a little bit more time -- it is difficult to quantify these costs because there is no baseline. You don't know how much the project would cost if you did not do all these things and you don't know how much more crappy the end result would be.

Boehm, B.W. 1976. “Software Engineering.” IEEE Transactions on Computer SE-1(4):1226-1241.

As cited in "The Economic Impacts of Inadequate Infrastructure for Software Testing" by RTI.

https://www.nist.gov/document/report02-3pdf

Pretty sure that both of those papers exist, and the NIST table cites up to 990x and refers to the 76 Boehm paper, not any IBM org.

Of course if you’re just running a website and you’ve got a nice network effect going on, or you’ve got your customers locked into contracts based on matters other than the users’ satisfaction all of a sudden matters of quality become a “cost” and you all of a sudden have to come up with this hokum to try and discourage your developers from doing a good job contra their intuition.

I have a close acquaintance who works in medlab and frustratingly we’ve seen this mentality creeping into the equipment even they use where the system can be down for days at a time due to some software update and then they won’t even properly resource field engineers because they then refuse to reap the costs that the 100x model predicts. Certainly they’re probably saving pennies on the dime but who ends up bearing costs but users and healthcare budgets.

There are those that will quibble this is but an anecdote but you can see this kind of sloppy crack-handedness creeping in everywhere. Oh ship it today we can fix it tomorrow when we ship a whole slew of new bugs too.

Sorry. A bit of a sideways rant.

I also just can't believe that any of these numbers are universal constants. For example, at my current job, a bug being discovered is likely to cause a moderate amount of damage, but actually shipping a fix for it might take months. At my previous job, I think my record time to have a fix for a defect deployed to production was about 30 minutes, but I also once had a bug do more than my annual salary's worth of tangible damage in just a couple minutes. At the job before that, production defects were typically relatively low-impact and also fairly inexpensive to fix.

It should come as no surprise that each of these companies had developed a dramatically different approach to quality assurance.

Where I'm going with that is, it's not just that the research is bad, it's also that our ways of trying to use that research are nonsensical. Even if that study existed and were accurate, it would still be useless for most of us. Because it would only describe how things worked under one set of conditions that is almost certainly not the set of conditions under which you are operating. And yet, people often quote it without offering any real context whatsoever, as if it were some fixed constant that applies to everyone everywhere.

It's not exacltly that the figure's veracity doesn't matter. Garbage in, garbage out. But if the thing you're putting stuff into and out of is something like a trash compactor, then the more immediately relevant principle is, anything in, garbage out.

If you find a bug during unit testing while developing a feature you just fix it.

If the bug is found in integration tests it may pull more resources and people to process. Even more so if found during integration tests.

If found after release to customer it might get very, very expensive.

The takeaway is to find and fix bugs as early as possible.

If you want to put an actual number on how much more expensive, you indeed need to carry a study but that will probably vary with the field or type of products and may in the end not provide additional actionable insights. Especially, I think this is risk mitigation so one would want to consider the worst case and its likelihood and not only an average number. That's probably difficult to capture in a generic study and product teams will know best what applies to their specific product.

Please note, that the axiom does not say "100x more", it says that cost increases (usually; it may even decrease, why not? consider a bug in ransomware that prevents it from encrypting files properly). The concrete cost may be 1.5x, 10x or 1Mx, it all depends on the circumstances.

And here's a pretty picture from the research paper on the subject: https://www.researchgate.net/figure/Cost-of-Fixing-a-Defect-...

I've fixed production bugs in literal minutes. Because the service was web API, that reported error as soon as that endpoint was hit. Sentry gave all the info I needed to know exactly what was wrong. Our procedures and tools for hotfixing made rolling out super fast.

Same bug in smartform app would have been days, and dependent on user upgrades.

Same bug in industrial or medical, etc proly months and much more dev effort due to testing verificatiin.

Point being bug cost has more to due with platform(web, app, desktop, embedded), ops environment(game vs lives at risk), and how much effort has been prespent in clean codebase, procedures, monitoring and tools. Than dev/prod environment.

If a study did exist and had the 100x figure based on real data, then it would still have been paraphrased enough to get back to the same status and no one would be aware of what the original conditions and assumptions were. So in many ways it doesn't matter if it exists and so it's non-existamce is not the problem that needs fixed, but how we document our knowledge generally.

The way I originally heard it was that problems are 1 x harder to fix during design, 10x harder during prototype and 100x during production. It's about orders of magnitude rather than the numbers. Another way of putting it is: problems get harder to fix the later you detect them.

If course there's a point of diminishing losses to aim for. Sweeping statements either way on software testing aren't helpful.

Is it cheap to detect error from initial requirement specifications or when the product is in sales and nobody buys it because of the requirement flaw?

If we make wrong kind of products in a way or another, it is always very expensive.

"If the cost of fixing a requirements error discovered during the requirements phase is defined to be 1 unit, the cost to fix that error if found during the design phase increases to 3 - 8 units; at the manufacturing/build phase, the cost to fix the error is 7 - 16 units; at the integration and test phase, the cost to fix the error becomes 21 - 78 units; and at the operations phase, the cost to fix the requirements error ranged from 29 units to more than 1500 units" [0]

[0] https://ntrs.nasa.gov/citations/20100036670

Now with most update and delivery mechanisms it should be studied.