Hacker News Reader

Comment by eloff | Hacker News Reader

There were only 12 people in the study, this can't have any statistical significance. The differences they saw could be completely caused by chance.

View on HN

> There were only 12 people in the study, this can't have any statistical significance.

That's just mathematically untrue. I'm not saying anything about this particular study though.

(For example, here's a study with 12 healthy people that'll get you a statistically significant result. Have 6 of them run a mile, and for the other 6, have them run a mile with their legs tied together. You'll find, with plenty of statistical significance, that having your legs tied together worsens mile run time performance.)

No, I stand by my assertion. You might have a "statistically significant" result, but the sample size is just too small to know if it represents a real result or not. At best you can say they've discovered something worth investigating with a larger study. But you can't take the conclusions at face value.

http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html

That depends on the study and what kind of results it gives and what the numbers turn out to be. While you're now agreeing that you do get so-called "statistical significance," the probability that you've gotten a useful result can also be high, too. You can even get one with a sample size of 1 -- Eddington's solar eclipse experiment is an example of that.

edit: Also, the possibilities for statistical significance and real useful results can be much higher when you have a highly dimensional signal.

Fair enough, and my bad for not using the term statistical significance correctly. But we agree that it's entirely possible to have statistical significance that's completely meaningless. Lies, damn lies, and statistics. This is especially true for health studies, and very common with studies that show so-called gender differences when what's really happening is the variation among individuals of a single gender is larger than the detected variation between sexes. Then a result with a small sample size is really just noise and an attempt to publish a paper for the sake of publishing a paper by some academics with questionable competence or morals (or both.)

Yes. I would expect the typical health study like this with 12 subjects to be weak. It's quite possible (possible) for this one not to be weak is because it involves DNA, which is a highly dimensional thing. For example, if they set out to look at the (large) set of genes with property S and see what proportion of those had property P for all women and not-P for all men or vice versa, and found a result like, 50% of these genes are like this...

At that point you know this made-up result is not luck, there absolutely is something different about these people. (But it is possible it just so happens all the men in the study ate Froot Loops as a kid, and all the women didn't, and that's the cause of all this.)

http://zedshaw.com/archive/programmers-need-to-learn-statist...

Reply on news.ycombinator.com