Hacker News Reader

Comment by MillenialMan | Hacker News Reader

The sample size was also 31 users, 40 controls. This really isn't a meaningful result.

I did not read the results, but this definitely depends on the effect size and the variance of whatever they are studying. If 31 people were raised from the dead, for example, it's plenty of sample size.

The issue is that the sample might be highly biased. If it's just 31 white males from Lubbock Texas then what does it even say? Stoners in that town speak different?

With such a small sample size, formal methods break down. The sample could be biased. In fact, it's almost guaranteed. The ability to p-hack (even inadvertently) is extremely high. Publication bias is also practically guaranteed. These kinds of issues void significance to a reasonably high degree.

At these kinds of sample sizes, the biggest issues are not going to be included or visible in the study. Unless the reported significance is extremely high, it's not meaningful.

I don't really participate in scientific literature or journals, but I see a lot of notes in HN around papers that appear to be overblowing the significance of their findings.

Is there a way for researchers to publish 'interesting observations' as just that, something that they measured that may or may not be noteworthy but felt worth sharing with the broader community?

The traditional solution to this problem is to insert a paragraph in the conclusion saying something along the lines of "this merits further study". However, that particular phrase has been overused in journal articles, to the point that it doesn't mean anything anymore.

It's not really something researchers can control. I was commenting on the news release, and it being posted on Hacker News, because that's more of an issue.

Reply on news.ycombinator.com