Comment by ericelliott | Hacker News Reader

Comment by ericelliott | Hacker News Reader

Parent Full thread

While the example is fictional, the numbers used for it come from a meta-study of TDD research, and for the relative cost of fixing a bug in production vs implementation time, "Integrating Software Assurance into the Software Development Life Cycle (SDLC)" from the IBM System Sciences Institute. Original sources and raw data are available if you click through the links.

None of the examples or figures presented rely on the Chrysler C3 project, or anything published by Kent Beck. I'm unsure why you'd mention them except to build a straw man argument. Straw men aside, this is all independent research with lots of 3rd party citations.

Yes, tests do take additional time and do incur maintenance costs as well, and that additional time is accounted for in the numbers presented. TDD still easily wins.

Yes, you can produce successful projects without TDD. You can also run a marathon with only 1 leg -- but maybe it would be easier with 2.

As for questioning my credentials, I don't blame you. It's certainly a lot easier to try to cast doubt on somebody's credentials than it is to come up with a valid argument or think critically about your own beliefs and bias. If you have any sincere doubts, a sampling of projects I've worked on are listed at the bottom of the article.

The linked article TDD: the Art of Fearless Programming by Ron Jeffries and Greg Melnik, IEEE Spectrum May/June 2007 https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf

cites Kent Beck's books Extreme Programming Explained (1999) and Test Driven Development: By Example (2002)

Kent Beck is Mr. TDD It is reasonable to question his claims about C3 as many have done.

The IBM article Integrating Software Assurance into the Software Development Life Cycle (SDLC) https://www.researchgate.net/publication/255965523_Integrati...

has the following section:

PROCESS TO SECURE CODE

In the event of a vulnerability finding, the software code may require redesign and implementation. This iterative cycle is costly in time and resources. To truly understand security threats to a system, security must be addressed beginning with the initiation phase of the development process. For an organization this means they must allow the IA controls and requirements to drive design and influence the software requirements. Therefore, any identified security threats found during the requirements and analysis phase will drive design requirements and implementation. Security defects discovered can then be addressed at a component level before implementation. The cost of discovery and mitigation can be absorbed within the review, analysis and quality check performed during the design, and implementation of our SDLC. The resultant product is one with security built in rather than security retrofitted. A study was performed by the IBM System Science Institute in order determine the relative cost in order to fix defects within the SDLC. Figure 2 displays their findings.

followed by a figure 3 (not 2) which is a simple graphic.

There is no identifiable reference to the underlying data which is presumably some sort of study for the US Department of Defense for security software issues presumably for military related software projects. This does not appear to be a general study, nor is it at all clear what was studied or how.

I am not questioning credentials but extrapolation from in Kent Beck's case a payroll system to general software development or in the IBM case from some sort of DoD software security projects to general software development.

DoD and aerospace projects often have unusually high costs for bugs in the production systems. A bug in a flight avionics system can cause a literal crash with loss of millions or even billions of dollars (Space Shuttle for example) and lives.

The costs discussed are counted in development time, not the monetary value of rockets. That point is clearly evident with a cursory glance at the math. Just look at the numbers. The values all fall on the same curve. The cost increase from design phase to implementation phase is the same relative increase from implementation phase to testing phase, which is the same relative increase from the testing phase to the production phase.

If it were counting the cost of rockets in the production phase, it would clearly jump that value off that cost curve in a dramatic and clearly notable way.

The mathematical evidence of these facts is clearly visible in this chart: https://cdn-images-1.medium.com/max/2000/1*jOS7CZX-gWfOoKGPU...

Which is a reproduction of the graph that appears in the original paper using identical data.

Of course, the data I cited was not the only data available on the economic impact of software quality. There is a whole chapter on the topic of the economic benefits of reducing bugs in the book "The Economics of Software Quality" by Capers Jones, Olivier Bonsignour Addison-Wesley, Jun 3, 2011.

The book is packed with valuable references if you're looking for more data on why it's worthwhile to reduce production bug density using techniques like TDD and code review. I strongly suggest you have a look at it if you have any doubts about the value of reducing production bug density.

Reply on news.ycombinator.com