Evidence-based Software Engineering [pdf] | Hacker News Reader

Evidence-based Software Engineering [pdf]

citeseerx.ist.psu.edu

12 pointsmblakele9 comments

the cornerstone of modern evidence-based medicine is the randomised trial. trials can produce statistically significant results because there are many humans and they are all quite similar. unfortunately, software projects are both more diverse and less common. this is a critical difference: you simply cannot replicate evidence-based medicine in the field of software development.

section 5.2.2 addresses some of this, but is then pretty much ignored in the conclusion.

Yes, that struck me too. Some developers do randomized trials of user experience, but such trials aren't tests of development techniques - unless perhaps you are working on an IDE.

And yet I think the points about negative evidence are relevant to software. Whenever a hot new trend surfaces, it seems like negative early experiences aren't reported. Later on there is usually a backlash, but it would be in our interest to shorten that cycle.

According to HN Guidelines (http://ycombinator.com/newsguidelines.html), next time please append [pdf] to the title. Thank you.

Done, thanks.

Well done. :)

This paper is from 2004, but I don't think I've seen it discussed here before. Any experiences or insights to share on this topic?

Greg Wilson did a book, which I did not enjoy, on this topic: http://shop.oreilly.com/product/9780596808303.do and a slideshow, which I recommend: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 .

My take is that there is much to learn from science about how to evaluate propositions regarding software engineering (most, but not all, of them are unsupported) but few new useful ideas.

Another reference along these lines: http://www.amazon.com/Facts-Fallacies-Software-Engineering-R...

I haven't yet read the paper but Joel of Joel on software fame, had discussed it once. See http://www.joelonsoftware.com/items/2007/10/26.html

Is table 1 really just doing a search-and-replace of 'clinical' for 'software engineering'??

Reply on news.ycombinator.com