It used to be assumed that differences among hospitals
or doctors in a particular specialty were generally
insignificant. If you plotted a graph showing the
results of all the centers treating cystic fibrosis—or
any other disease, for that matter—people expected that
the curve would look something like a shark fin, with
most places clustered around the very best outcomes.
But the evidence has begun to indicate otherwise. What
you tend to find is a bell curve: a handful of team
with disturbingly poor outcomes for their patients, a
handful with remarkably good results, and a great
undistinguished middle.
http://www.newyorker.com/archive/2004/12/06/041206fa_fact?cu...Statistics requires a lot of correct assumptions in order to be accurate; sadly, most people overlook or fail to check their assumptions.
The problem is people assuming a normal distribution on a single variable with unknown distribution.
[1]: http://en.wikipedia.org/wiki/Central_limit_theorem [2]: http://stats.stackexchange.com/questions/22387/why-it-is-oft...
We're not talking about the statistical central limit theorem here, we're talking about people who are passionate about programming and spend years working on their craft.
You can only invoke the normal distribution thing when you're talking about an outcome that is the average of many independent quantities.
Which is not to say there is no market for chasing frisbies.