Comment by SloopJon | Hacker News Reader

Comment by SloopJon | Hacker News Reader

Parent Full thread

I don't recall encountering this specific 100x claim, but I do recall Steve McConnell making a similar case early in Code Complete. In chapter three, he has a chart showing that the average cost of fixing defects post-release is 10-100 more expensive than during requirements, which he supports with the following citations:

"Design and Code Inspections to Reduce Errors in Program Development" (Fagan 1976)

Software Defect Removal (Dunn 1984)

"Software Process Improvement at Hughes Aircraft" (Humphrey, Snyder, and Willis 1991)

"Calculating the Return on Investment from More Effective Requirements Management" (Leffingwell 1997)

"Hughes Aircraft's Widespread Deployment of a Continuously Improving Software Process" (Willis et al. 1998)

"An Economic Release Decision Model: Insights into Software Project Management" (Grady 1999)

"What We Have Learned About Fighting Defects" (Shull et al. 2002)

Balancing Agility and Discipline: A Guide for the Perplexed (Boehm and Turner 2004)

I only have one of these (Dunn), and it's boxed up in the attic, so I can't readily check their sources, but I somehow doubt that all of these simply launder the study under discussion.

I don't want to trivialize some of the good points that Hillel Wayne makes about soft-science research applied to software productivity, but he would have us dismiss all of these citations out of hand, simply because they predate capital-A Agile, which of course changes everything. That doesn't strike me as a particularly compelling approach either.

> … he would have us dismiss all of these citations out of hand…

Does he? I thought the point of his talk [1] was that we developers ascribe way too much weight to few, small studies. So rather than saying that we should dismiss the claims, we should instead take great care because the claim may not be generalizable at all.

[1] https://www.hillelwayne.com/talks/what-we-know-we-dont-know/

This is what I'm referring to:

> A lot will be from before 2000, before we had things like "Agile" and "unit tests" and "widespread version control", so you can't extrapolate any of their conclusions to what we're doing. As a rule of thumb I try to keep to papers after 2010.

While I admit that even in the late 90s, it seemed strange to me to see citations from the 70s or 80s about projects from the 60s (like, say, OS/360), I'm not convinced that so much has changed in the last ten to twenty years as to render all previous research irrelevant.

Yeah I'll admit it was an off-the-cuff quip that really isn't all that accurate. I don't put as much time into editing the newsletter as I do into my proper essays, so stuff that I'd normally polish out gets through. I do prefer to keep to papers after 2000 in general, less because of dramatic quality differences and more because it leaves fewer ways for people to dismiss stuff without looking at it.

It's also bizarre to see claims that unit tests are so new. I can't say I really know about other communities, but Perl at least was doing a lot of unit testing using the Test Anything Protocol (TAP) back in 1987.

https://en.wikipedia.org/wiki/Test_Anything_Protocol#History

If anything it raises a much better question that a survey of past research might help answer, is there a difference in productivity over the last 20 years since Agile/source control/unit testing became popularized?

I enjoyed the book _Leprechauns of Software Engineering_ which did track down all the chains of citations to find where the original work was misunderstood or even nonexistent. I would bet that it covers most or even all of these citations, but I’m not taking the time to pull it out and cross reference. https://leanpub.com/leprechauns

One of the most recent citations (Shull et al. 2002) is freely downloadable:

https://www.computer.org/csdl/proceedings-article/metrics/20...

Its own citations may not be satisfying, but I find it nevertheless interesting. Here's the summary of the eWorkshop discussion of "Effort to find and fix":

> A 100:1 increase in effort from early phases to post-delivery was a usable heuristic for severe defects, but for non-severe defects the effort increase was not nearly as large. However, this heuristic is appropriate only for certain development models with a clearly defined release point; research has not yet targeted new paradigms such as extreme programming (XP), which has no meaningful distinction between "early" and "late" development phases.

Yes, and it’s worth noting that Hillel was referring to _Leprechauns_ when he talked about the cost to fix claim.

Reply on news.ycombinator.com