Hacker News Reader

Comment by pdar4123 | Hacker News Reader

Pro tip - When “every single one of the 96 participants show[s] the effect“ u have a technical artifact. This is scientific common sense.

View on HN

Not really. If you are counting fingers and toes you would not be surprised if all 96 participants had 10 of each. It is all about context.

It is indeed about context, and the context is front-line research in an established field - so the chance that your effect is as prevalent as the standard number of fingers an toes in humans and yet undiscovered is what should give you pause.

But you would be surprised if they all had 11, 10 is your null hypothesis. It’s not context, it’s proper controls and the scientific method. This was very very careless

Your rebuttal stands but not your point, for the comment you were replying to didn't properly contextualize their argument in a scientific experiment. It's more like serve participants canned vegetable soup, but in the non-control group warm it up before serving. Here you'd reasonably expect the non-control to be universally more favorable.

They did have a null hypothesis and a control group. The problem was that the non-control was different from the control in a way they were careless with, that the scientific methodology does not catch (that peer review and replication ought to have caught). e.g. if the favorable results were better explained by subjects seeing soup taken out of the pot vs. from the can rather than the temperature itself.

> If you are counting fingers and toes you would not be surprised if all 96 participants had 10 of each.

It is absolutely possible to find people with an abnormal number of fingers or toes.

https://en.wikipedia.org/wiki/Oligodactyly

https://en.wikipedia.org/wiki/Polydactyly

If 100% of the studied humans have 10 fingers and 10 toes, something's wrong with the study. In this case, the problem is the extremely small sample size: it simply doesn't represent the general human population. In the article author's case, a programming error caused these results.

0% and 100% are always suspect no matter the context they appear in.

You're nitpicking the example, rather than responding to the underlying point.

Whenever my results seem too good, I do a whole new round of checks above my usual. That said, I've had some real ones with effects that large...its just usually that effect is known already and not of immediate interest. Large effect sizes in psychological sciences usually means it has been discovered already :)

Reply on news.ycombinator.com