The Outrageous Cost of Skipping TDD and Code Review
medium.com
medium.com
Of course, this is a fictional example making a lot of assumptions. Your mileage will vary based on lots of factors, including your team’s experience with TDD, the presence or absence of other quality measures (such as code review), etc…
Here are the assumptions made:
Production bugs without TDD: 100
Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%)
Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost)
Average time to fix a production bug: ~15 hours
The relative results of this fictional example ring true to me based on my experience with real production projects, both with and without TDD.
BUT...
Where is the specific data and how representative is the author's experience of all or most software development?
Software development is enormously varied. Kent Beck initially hawked TDD based on a project to rewrite Chrysler's payroll system that was cancelled.
https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
Also see:
https://stackoverflow.com/questions/10149798/how-tdd-is-rela...
The Chrysler C3 project is not the most convincing argument for TDD or other of Beck's ideas. But even assuming the C3 project was a roaring success -- a highly debatable proposition, one should ask how applicable the results are to many other quite different types of software projects.
Similarly, what is the author of this article's actual experience and how applicable is it to other types of software projects?
I don't question that automated tests can be helpful and indeed critical in some projects but I do question rigid orthodoxies like TDD or requiring unit tests for every function or class after implementation (many people promote unit tests everywhere while claiming they are not doing TDD).
Tests take additional time to design and implement and thus cost more. They also have maintenance costs. Architectural and other changes in the software as it is developed can and do result in revising or rewriting the tests as well.
Many successful software projects were designed and implemented in the 1970's, 1980's, 1990's, and even today without TDD and without unit tests -- or code reviews although I am focusing here on TDD and unit tests.
None of the examples or figures presented rely on the Chrysler C3 project, or anything published by Kent Beck. I'm unsure why you'd mention them except to build a straw man argument. Straw men aside, this is all independent research with lots of 3rd party citations.
Yes, tests do take additional time and do incur maintenance costs as well, and that additional time is accounted for in the numbers presented. TDD still easily wins.
Yes, you can produce successful projects without TDD. You can also run a marathon with only 1 leg -- but maybe it would be easier with 2.
As for questioning my credentials, I don't blame you. It's certainly a lot easier to try to cast doubt on somebody's credentials than it is to come up with a valid argument or think critically about your own beliefs and bias. If you have any sincere doubts, a sampling of projects I've worked on are listed at the bottom of the article.
cites Kent Beck's books Extreme Programming Explained (1999) and Test Driven Development: By Example (2002)
Kent Beck is Mr. TDD It is reasonable to question his claims about C3 as many have done.
The IBM article Integrating Software Assurance into the Software Development Life Cycle (SDLC) https://www.researchgate.net/publication/255965523_Integrati...
has the following section:
PROCESS TO SECURE CODE
In the event of a vulnerability finding, the software code may require redesign and implementation. This iterative cycle is costly in time and resources. To truly understand security threats to a system, security must be addressed beginning with the initiation phase of the development process. For an organization this means they must allow the IA controls and requirements to drive design and influence the software requirements. Therefore, any identified security threats found during the requirements and analysis phase will drive design requirements and implementation. Security defects discovered can then be addressed at a component level before implementation. The cost of discovery and mitigation can be absorbed within the review, analysis and quality check performed during the design, and implementation of our SDLC. The resultant product is one with security built in rather than security retrofitted. A study was performed by the IBM System Science Institute in order determine the relative cost in order to fix defects within the SDLC. Figure 2 displays their findings.
followed by a figure 3 (not 2) which is a simple graphic.
There is no identifiable reference to the underlying data which is presumably some sort of study for the US Department of Defense for security software issues presumably for military related software projects. This does not appear to be a general study, nor is it at all clear what was studied or how.
I am not questioning credentials but extrapolation from in Kent Beck's case a payroll system to general software development or in the IBM case from some sort of DoD software security projects to general software development.
DoD and aerospace projects often have unusually high costs for bugs in the production systems. A bug in a flight avionics system can cause a literal crash with loss of millions or even billions of dollars (Space Shuttle for example) and lives.
If it were counting the cost of rockets in the production phase, it would clearly jump that value off that cost curve in a dramatic and clearly notable way.
The mathematical evidence of these facts is clearly visible in this chart: https://cdn-images-1.medium.com/max/2000/1*jOS7CZX-gWfOoKGPU...
Which is a reproduction of the graph that appears in the original paper using identical data.
Of course, the data I cited was not the only data available on the economic impact of software quality. There is a whole chapter on the topic of the economic benefits of reducing bugs in the book "The Economics of Software Quality" by Capers Jones, Olivier Bonsignour Addison-Wesley, Jun 3, 2011.
The book is packed with valuable references if you're looking for more data on why it's worthwhile to reduce production bug density using techniques like TDD and code review. I strongly suggest you have a look at it if you have any doubts about the value of reducing production bug density.