Hacker News Reader

Comment by parton | Hacker News Reader

If you were to ask experts in a given subfield which papers are reliable, I'm sure they would be able to tell you. The problem is that there's no process in science for expert consensus to make it to out to doctors/laypeople.

People assume that peer review means a paper is good, which couldn't be farther from the truth. Science journalists aren't any better, they care more about hype than consensus. Honestly, it's dangerous to give a random peer reviewed article to someone who doesn't have broad knowledge of the field.

Maybe we need middle-ground journals that publish review articles at the level of a Scientific American reader?

View on HN

> People assume that peer review means a paper is good ...

"Conclusions: Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention."

https://www.bmj.com/content/363/bmj.k5094

see the "Peer review" https://www.bmj.com/content/363/bmj.k5094/peer-review

Please tell me this is satire

From the first review:

>"While this trial illustrates many important points about participant screening and recruitment in RCTs along with the need for equipoise, we feel it is important to emphasise that this is not intended to undermine the use of RCTs, but to illustrate some of the issues that may arise.

We felt another important message from this paper is that if you make a decision based on abstract alone, this may lead you to an incorrect assumption, so this trial illustrates very well the importance of reading the whole paper."

The abstract is quite explicit about the limitations of the study, namely, that all the people without a parachute jumped from a very low altitude (mean of 0.6m) and that the results should not be extrapolated to jumping out of planes which are much higher than that..

The BMJ Christmas issue is full of funny (but maybe also serious) articles each year. My favorite is an analysis of reported virgin births [1].

[1] https://www.bmj.com/content/347/bmj.f7102

The BMJ is a very real and serious. This article is from the “Christmas Issue” which is a little different…obviously.

It is meant to be lighthearted and tongue in cheek - but no less serious of science. It often exists as a form of satirical critique of bad habits of science and science communication.

There are flaws in that model though that have been noted. Prime among them is that nothing clearly identifies an article as from the Christmas issue online, and even if it did that may not carry any meaning for a reader unfamiliar with the journal.

https://www.bmj.com/about-bmj/resources-authors/article-type...

It’s not a given experts know what papers are reliable. Here’s a paper from Genentech https://pubmed.ncbi.nlm.nih.gov/15385631/ with 500 citations which my advisors swore is reliable because “they trust the authors.” Don’t mind the fact that the central thesis of the article is barely supported by the last figure in which they conveniently average data from completely different sets of experiments for each time point.

And did I mention my advisors released a drug into the market just recently? Lol.

Journals could open replication wings? I'd rather read about experiments that have been validated by an independent third party at this point.

Part of the problem is the lack of prestige in replication. What gets researchers promoted is finding novel methods, so most researchers don’t want to spend their precious time on replication.

There is prestige in what gets you recognition, so if journals and departments started recognizing contributions to replication research, like giving them dedicated space in journals, then the prestige will follow.

Fair enough, but the incentives don’t align. Novel research attracts funding, so every if replication gets prestige in the future the business side of research will still run counter to it. A researcher would still gain more prestige work novel work that attracts substantial funding.

Most novel findings don't replicate, so a researcher at Yale can gain plenty of prestige by debunking the sensationalized findings from Harvard, for instance.

>Most novel findings don't replicate

This is very context dependent so such a broad generalization probably goes too far. It seems like the “soft” sciences have a much bigger replication crises.

But that point aside, your Yale vs. Harvard example doesn’t fix the funding issue I alluded to.

That’s pretty bang on (biochemist).

That's one of the intended purposes of review journals like Annual Reviews (https://www.annualreviews.org/). There are some pretty big practical issues with them though:

1. Sometimes the latest article on a subject was written 2/5/10 years ago. Depending on quickly the field is moving, that review could be perfectly acceptable, completely obsolete, or anywhere in between.

2. Sometimes the author themselves is just out of date with the field. It's difficult to identify this without deep prior familiarity.

3. Paywalls.

I don't know what the solutions are beyond "making experts accessible for questions" and other public outreach things.

Reply on news.ycombinator.com