On the other hand, what they are trying to study is incredibly complex - not just people, not just software, but people working on software, and even social systems of people working on software. How do you even begin to reliably measure that? Psychology looks like physics by comparison. Yet the resources available to do these studies are a drop in the bucket. This is not something society values – it's not even something our own industry values. So it's unsurprising that what we have so far is at best a smattering. You can argue that these toy studies ought not to be dressed up in formal scientific wear, and I don't disagree – it gives an altogether misleading impression of how solid their conclusions are. But no doubt that's true of a lot of papers.
(Oh and on top of all that, much of the literature is hard to track down. Didn't anyone notice that the OP wasn't able to even find half of the citations he was trying to check? Based on that alone he shouldn't be claiming that "the table is completely false", only that he was unable to rebuild it from data.)
The real question is whether we should throw it all out as junk or try to make something of it. I think there's a good argument for junking. But it's also reasonable to say: hold on, for all their weaknesses these studies are all we have and they're not nothing. At least someone is trying to work with actual data. So let's consider them, but carefully. Like walking on a swamp.
McConnell is one of the few people who have made serious efforts to walk the swamp and filter it somewhat and convey its findings to professional programmers. Of all these people (that I've read), he actually stands out as the most meticulous. To fire cowboy accusations of dishonesty at him personally is likely unfair and misses the point. I do think McConnell overestimates how solid the swamp is, but he's not ignorant about it (read that whole post) and the claim that he outright falsifies citations, or anything like that, demands a high burden of proof. I haven't read Bossavit's critique (should I?) but anyone taking that position had better first make sure he's not standing on a swamp too. That's harder than it sounds.
Author of "Leprechauns" here.
I'm not sure I follow you in the above sentence. What happened time and again in my investigations was that I would find a citation (in McConnell or in Boehm or elsewhere) that was metaphorically accompanied with the statement "this here is a solid piece of land in the swamp".
When I got there, however, what I found was, in fact, just more swamp.
It takes a higher standard of proof to demonstrate that something is solid ground than to demonstrate there's a flaw in it. That may be unfair, but it's why scientists need a lot of training.
To take just one example, Graham mentions the Hughes Aircraft study, cited (by McConnell and others) in support of the usual "exponential" curve for defect cost (as "Willis, Ron R., et al, 1998").
When you actually read that paper - which is both one of the easiest to obtain and one that has the more detailed data about what was studied and how - and look for the raw data, you find numbers that do not obey an exponential law OR a monotonic increase. For one column for instance the numbers in fact vary within a narrow range .36 to 2.00 with two maxima at the "Coding" and "Functional test" phases, dropping off before, between and after. In a different column the costs vary only by a factor of two between Unit Test and System Test, by a factor of less than four between Code and System Test. The exponential rise is generally not true for the pre-1991 period; some post-1991 measurements get closer, but many do not.
It's hard for me to see this evidence as doing anything else but undermining the original claim of the exponential rise as a generally reliable regularity in software development.
Given this, I don't think that citing this paper in support of the claim is appropriate.
I'm saying that if one is going to attack someone personally for dishonesty, as opposed to critiquing the state of the field in general, the bar for that is high and one had better be more than careful with one's own particulars. Reasonable people can interpret this stuff differently. The post linked to upthread [1] didn't strike me as dishonest (though of course it's only one side).
I'm glad you've been looking for solid ground in the swamp and finding it swampy. I think that's valuable. What bothers me are the hints in the OP, comments here, and elsewhere I've seen (including people talking about your work) that this is about one guy making shit up when in reality the problem is endemic to the entire field. Unless he's been egregiously dishonest, which I doubt, it's a distraction.
On another note, you seem like a good person to ask: is there any finding in the software engineering literature that you think holds up? i.e. have you found any solid ground in the swamp? I'm not sure I have (but I haven't looked nearly as hard as you). If there really isn't anything, that alone is kind of shocking.
[1] http://forums.construx.com/blogs/stevemcc/archive/2011/01/09...
Hackers hate ad hominem, and with good reason. I too subscribe to the school of "harsh on the problem, soft on the person". On the other hand, it makes no sense not to call out things that keep us in the swamp, or to tiptoe around important epistemic issues just to spare hurt feelings.
One problem we have is that few people are willing to go to great lengths to check out the available evidence; instead the pattern is to repeat claims (and associated citations) made by people who sound authoritative, accepting them essentially on faith. This has the unfortunate side-effect of magnifying the mistakes of people who have become authorities.
In "Leprechauns" and elsewhere, my focus isn't on what any particular person says, but on specific claims. "The cost of fixing defects rises exponentially as a function of SDLC phase" isn't tied to a particular person - it originated with Boehm but many others have propagated it. My method has been to look up the evidence and to see if it held up. Also to think hard about why studies may have failed to show conclusively what they set out to prove, and how to overcome these challenges.
I'm doing the same kind of thing in my own area, i.e. I'm collecting all available evidence, pro or con, about whether various Agile practices "work".
> is there any finding in the software engineering literature that you think holds up
There are many good ideas and rules of thumb, but when it comes to very general "laws", solidly established - that's harder. I've been asking that question over and over again, hoping to get a convincing answer. Still haven't got one.
I've read a bunch of supposedly solid references, e.g. Pressman, or the "Handbook of Software and Systems Engineering" and have been underwhelmed. (The first "law" proposed in the Handbook: "Requirement deficiences are the prime source of project failures," based on evidence like the Chaos Reports. My rebuttal: http://lesswrong.com/lw/amt/causal_diagrams_and_software_eng... )
There does seem to be a recent wave of software engineering literature, exemplified by http://www.amazon.com/Making-Software-Really-Works-Believe/d... (which I haven't read). Are you familiar with this more recent stuff? Does it represent new research or merely new reporting on old research? If the former, are the standards higher?
You might be able to tell why I lost my enthusiasm for the book.
I think that's the argument for junking the SE literature. If it can't do any better than anecdote, well, to quote Monty Python, we've already got some they're very nice.
"The fact that no studies have produced findings that contradict the existence of a teapot orbiting Jupiter..."