Comment by matthewdgreen | Hacker News Reader

Comment by matthewdgreen | Hacker News Reader

Parent Full thread

There is a major issue of limited resources to replicate results, in terms of both time and funding. For example: I would assume that most important results are replicated. As a concrete example, if someone identifies a medication that (in one small trial) shows a statistically significant effect in curing some serious medical condition, then this will drive further replication attempts. On the other hand, if someone publishes a study showing that holding a pen in your mouth makes you 1% likelier to do well on the PSATs, this study will probably languish without replication for a decade — because honestly who cares? It’s basically a curiosity. I can’t help but notice that many of the headline results that characterize the “replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news.

I’m not saying that our current allocation of resources is optimal. I am pointing out that our resources are finite and “replicate everything” is not even a remotely practical allocation of those resources.

> I would assume that most important results are replicated.

GP is pointing out that the incentive structure makes this an invalid assumption. If publications reward hype and novelty when deciding what to publish, then there is no point spending your limited resources replicating other peoples' results, they won't get published anyway. And experiments that give a null result won't be published anyway. What's left are one-off results that showed something surprising simply by chance and don't replicate...but then, we generally will never know that they don't replicate, because the replication experiment is not novel, has a low chance of being published, and hence isn't worth spending limited resources on.

Basically the publication process introduces selection bias into the types of research that are even attempted, which then filters down into the conclusions we take from it. A cornerstone of the scientific method is random sampling, but as long as the results that get disseminated are chosen by a non-random process, it introduces bias.

>“replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news.

I don't know that this is accurate. Some of these make their way to large-scale public policy, or give bona fides to people who craft far-reaching policy. This includes changes to 401k allocations to car-insurance rates and other mundane, but consequential policies.

The truth is most of science is not important outside of popular science news. So we shouldn't be surprised that the bulk of replication crises are also in the same category. Claiming this means the replication crises is not really impactful may be a case of base rate neglect.

It's also important to note that your example of medical replication is a relatively highly regulated area, where most other science is much less so.

> For example: I would assume that most important results are replicated.

The example you provide is solely your assumption? Seems pretty odd to provide a baseless assumption as an "example".

> There is a major issue of limited resources to replicate results, in terms of both time and funding.

Undergraduate students love being involved in research. It's one of the selling points of many top universities. Grad students replicate research all the time. Maybe funding is an issue in grant fields (some research is extraordinarily expensive) but that doesn't excuse the lack of replication across the board.

I’m a researcher at an R1 and I love working with undergraduates. But it’s not true that they’re a huge source of additional resources. Quite the opposite: like any new researchers they have to be carefully supervised, which means they can actually consume a lot of time from PIs and PhD students (you can send them off on their own without supervision, but that’s how you get low quality results.) Unlike PhD students, however, undergrads tend to graduate just as they’ve finally learned the ropes. It’s still worthwhile because a few of my undergrad research interns did great work, and at least a couple of them applied to PhD programs (at least two in my lab; they’re both Assistant Professors at top CS departments.) But the answer to a resource allocation problem is almost never “here’s a simple idea that makes the resource limits go away, why aren’t people doing that.” Unfortunately.

Reply on news.ycombinator.com