Hacker News Reader

Comment by asQuirreL | Hacker News Reader

After reading the paper [1] regarding the empirical study on TDD, I'm pretty dubious about how well backed the conclusions are. If you look at Table 2, in the 4 case studies, 2 of the projects took longer with TDD (2.00x, and 2.64x ratio in man months to 3sf) but then the other 2 did better: (0.319x and 0.476x ratio in man months, 3sf). But all of this seems to be ignored in proving the original statement that TDD take 15-35% longer, which appears to be backed by what is described as "Management Estimates" [Table 3].

In fact, taking the geometric mean of the man month values (which, I think, but am not sure is the right metric to use when comparing ratios), we find the average ratio to be 0.947x. Make of this what you will, given the size of the sample.

[1]: http://research.microsoft.com/en-us/groups/ese/nagappan_tdd....

View on HN

The Nagappan paper gets cited a fair bit, mostly because there is so little hard evidence of any sort that actually backs TDD as presented by the big name evangelists.

However, on a closer reading, even that paper isn't really assessing TDD. It basically defines any test-first coding practice to be TDD, including those that still spend considerable time and resources on up-front design.

What we really need is some properly structured study that compares professional practitioners (not just students or recent grads) over extended periods (not just a brief project that doesn't show the long-term benefits or costs in terms of quality and maintainability) who are using no unit tests but otherwise common development practices (control group), the same but writing unit tests after the main code, the same but with unit tests written before the main code, and full test-driven development where other aspects of the development process, particularly design, are effectively replaced by the test-writing activities as advocated in well-known TDD books and other training materials.

Of course it's all but impossible to really achieve that sort of like-for-like comparison with usefully controlled conditions, which is why these issues are so hard to debate objectively. But try finding any study with pro-TDD conclusions that doesn't obviously fail the objectivity test in at least one of the three ways above (not a sample of professional programmers, too short term to draw general conclusions, or conflating TDD with some other unit-test-related development process).

You're right, this kind of study is really hard to set up properly. However, in this instance, I'm more concerned by the quality of the analysis. It seems to be, ignoring actual data in favour of anecdotal evidence, even when they clearly aren't even correlated at a cursory glance.

It would be really interesting to see how much variation there was between two groups using the same method as well, to get an idea of how much different team vary, and how much controls can be relied on in such comparisons.

Reply on news.ycombinator.com