Hacker News Reader

Comment by 0cf8612b2e1e | Hacker News Reader

I am not sure you appreciate the scope of distrusting submissions. Does this mean the reviewer has to clean-slate replicate everything? What if the author spent three years breeding a particular mouse model? Purified 8pg of an antibody extracted from flea tears? Spent six months of HPC time to simulate molecular dynamics? Some science is hard and incredibly difficult and expensive to do.

Even just reevaluating a submitted dataset is no guarantee. Someone committing an elaborate fraud could have faked the raw datasets as well.

View on HN

> I am not sure you appreciate the scope of distrusting submissions.

What you're saying is, the first option I gave, to fix the peer review process so it is a good indicator of the reliability of research, is not feasible.

If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.

That's actually exactly correct. Peer review is supposed to characterize the research in terms of quality, not validate it. Otherwise, those peers who validate it would practically be co-authors.

It's a system built upon honesty. A dishonest person is obviously going to take advantage of it. Really blatant things are caught in peer review, but sneaky fraud will definitely slip through.

> Peer review is supposed to characterize the research in terms of quality

Fraudulent research would obviously not be of the desired quality, wouldn't it?

> It's a system built upon honesty.

But if the purpose is to characterize the research in terms of quality, this can't be right. Since fraud is known to happen, and there are known to be incentives for fraud, any process that claims to judge the quality of research would have to have some way of checking for fraud.

I think you're misunderstanding what I mean by "quality" in this context. It's not quality as in something like meat, it's quality as in "a quality education". Does it go above and beyond vs. do the bare minimum to address the scientific question proposed by the authors.

Something can appear to be of very high quality and yet still be completely fraudulent. This occurs in countless fields; art, counterfeit money, deepfakes, etc.

Scientists are attempting to address fraudulent research in other ways, but peer review is not the best means for it and it should not be thought of in that way.

> Something can appear to be of very high quality and yet still be completely fraudulent.

So in other words, peer review does not judge actual quality. It only judges apparent quality. Which is not what anyone actually wants to know.

> If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.

It is the case, and that second option is exactly what we do. One article is nice, but replication is key before some new finding is accepted. That is exactly what happened with LK-99, for example, which is interesting because it was very public.

> that second option is exactly what we do

I don't think so. I see claims made all the time about science being "settled" purely on the basis of it being peer reviewed.

> replication is key before some new finding is accepted

If only that were true. In many cases it isn't.

Reply on news.ycombinator.com