Hacker News Reader

Comment by jfmc | Hacker News Reader

I guess that 100x is just an arbitrary number that means 'orders of magnitude'. If production means going to Pluto on a super expensive space mission, and the bug ruins all the scientific experiments, and your future funding depends on the success of the mission, I'd say that the cost tends to infinity. If the product requires some kind of polishing that only interaction with real users can provide, then the cost of bugs at production be even negative (spend less time on the requirement side, etc.).

View on HN

You need to remember the context when this study supposedly took place. "Production" used to have a literal meaning -- producing physical media. 100x was a reasonable number to believe in that case even without extreme situations like a mission to Pluto.

I first learned this in QA training. It's been decades so I don't remember if 100x was mentioned. They listed all the people required to fix a bug once it had gone to production. Off the top of my head that included:

- Customer support handling calls from customers -- possibly thousands of times. This isn't necessarily part of the cost of fixing the bug, but it's part of discovering a bug in production.

- Some combination of managers and product managers prioritizing the bug and scheduling programmers to fix it.

- Programmers finding and fixing the bug.

- QA validating the fix.

- Writing an installer for the fix.

- QA validating the installer.

- Producing the fix.

- Shipping the fix to effected customers -- which may be all customers.

We were being trained to do QA, so we compared that to the cost involved when QA discovers a bug. That was essentially just the cost of us writing a bug report and programmers fixing the bug.

Adding onto this, if the fix requires some kind of notice or error message detail that didn't previously exist it can also require work from your internationalization team to translate that error into N languages your software supports.

I did forget that. I'm pretty sure that was actually listed in the training because the company I was at made software that was translated into a lot of languages.

Replying to myself because I got too excited and hit enter.

Compare the above to a modern SaaS company. Customer support handling the reports is a fraction of the cost because it doesn't require spending time on the phone with each customer. Most don't have QA, so those costs are gone. No writing and validating installers. No cost of making and shipping physical media.

One other thing is that programmers were a lot cheaper then. So the time a programmer spent actually fixing the bug was even less significant.

The costs are not gone, they became externalities. If you're a SaaS customer, and wrote some code that (unwittingly) depends on the bug, now you have to rewrite it, for instance.

There's a more subtle issue with fixing bugs once you have shipped: Some customers might consider the bug as part of the product and built functionalities around it.

https://xkcd.com/1172/

https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost...

Learned of this one [0] the other day, in the remarks of System.Net.IPAddress.TryParse:

"Fixing this bug would break existing apps, so the current behavior will not be changed"

In other words, the obvious method to use to validate whether or not a string contains a valid IP address can't be used because existing applications expect the broken behavior, and thus it will not be fixed. Correct parsing of an IP address is an exercise now left to the user (to get wrong).

[0] https://docs.microsoft.com/en-us/dotnet/api/system.net.ipadd...

100x is probably conservative in an enterprise pure waterfall development environment, regardless of application.

This is why so many developers can’t be left to their own devices to decide when a product is launch ready. There will always be bugs. Always. You can ship with known bugs. It’s about being pragmatic and fixing it at the right time. Not all bugs have the same impact.

You could make the exact argument about product managers..

This is why product managers / business can't be left to their own devices to decide when a product is launch ready. Not all bugs have same impact, and they are not experts to understand which are high impact.

This statement is just as true as yours. Doesn't mean you should ignore either one group to make a decision.

Ha! you're being too kind. :-)

How about life saving medical applications (insulin pumps, pacemakers etc), infrastructure (power, transportation, internet) let alone weapons of mass destruction. Then, consider subtle security holes as serious bugs to be exploited en masse. People try to air gap the hardware, and even that's not always enough (see StuxNet).

Yeah, but this is the point. It really depends on the context.

Trying to fix every bug in the pure informational website but missing your product launch? Not worth it, just fix it later. (or not at all, have you surfed the web with the console open? It is flooded with warnings and errors)

But miss the critical bug in the webshop, that cause customers to legally buy for free? That is expensive.

As a software engineer who has been mostly focused on testing over the last few years and leading a team focused on this, it's clear to me now that balancing what's worth testing and what's not is a delicate art and very difficult to always get right.

I do believe that a common failure is not investing in adequate regression focused tests. I've been on projects were these were not in place and an update lost a company millions of dollars very quickly prior to being noticed in production, I've been on projects were delaying any further to release would have lost critical learning data during a period of time that could not be guaranteed to come again quickly (for example market volatility or turmoil).

The best engineers in my opinion love testing as much as building new features because they understand that production fire fighting is the least fun activity of all.

Quick plug, I believe solid test reporting makes the ROI on test development clearer. I'm founder at Tesults - a test results reporting app (https://www.tesults.com) and I'd love your feedback on it, send me an email if you have a moment.

Even if 'it's orders of magnitude' it still must be orders of magnitude cheaper than for some other industry. I bet recalling millions of cars due to some small mechanical bug is very expensive.

Or see the case of the installer on Myth 2, that you could also say the cost was infinite, since the but on the installer made the company litearlly go bankrupt and get purchased by Microsoft.

1. It was the uninstaller, not the installer.

2. That is not at all what happened. Bungie never went bankrupt.

Reply on news.ycombinator.com