Hacker News Reader

Comment by cjbprime | Hacker News Reader

None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.

View on HN

I kinda agree, but at the same time this seems like an exceedingly pessimistic take. Maybe this can be the devil on one shoulder while you have an angel on the other pointing out that papers already go through peer review, that overt fraud is rare, and that the biggest problem we have isn’t bad science, it’s taking scientific results out of context, misreporting them, overgeneralizing the results, and painting lines where none exist. The actual observations in a paper are usually real and legit, the problem is that humans seek to explain those observations as a pattern, and describing that pattern and assuming it even exists, that’s where we fail so often. More often than the paper’s conclusion being wrong, people cite a paper as evidence for something “related” or “similar”, where the paper’s observations don’t actually apply.

FWIW, I think you're actually being insufficiently pessimistic, and also the example effects I mentioned (publication bias and multiple hypotheses) exist in the zone where both are true -- the researchers observed something, and also the research claims are false.

Peer review is regularly completely inadequate. As an example, consider the amyloid hypothesis for Alzheimer's, which just this week was likely discovered to be actively fraudulent after nearly 20 years, thousands of papers, and thousands of scientists spending their entire research careers on it.

Again, while I think those things may be true sometimes, they aren’t the biggest problem we have in terms of science. And I feel like choosing to frame it the way you’re framing it is intentionally ignoring how often things go right, how often people are sincere and honest and competent and right.

We already know peer review isn’t perfect, but it’s also probably a lot better than the layman’s attempt to discriminate good science from bad. We don’t have a ton of data of how many papers are rejected either. But, one high profile example isn’t a particularly scientific way to establish that all of science is actively bad. It’s one bad example, and yes it’s awful that one case can have such wide ranging ramifications, but you’re downplaying the thousands of papers, and thousands of scientists who assumed the research was valid and did good faith work on top of it.

If anything, the practical reaction to your example should be that papers that go high profile need replication studies very quickly, not that we should assume as a lay person that all science is wrong.

Overt fraud is rare, but sometimes the line of fraud is worryingly thin. Sure, making up data is fraud, but what about stopping your experiment early? Discarding some perfectly good data points? Tweaking your model until it just falls below p=0.05? These are all real things that are depressingly common in real studies published in prestigious journals. Sketchy correction factors, unsound data collection methodologies, baffling experimental design, the list goes on and on.

Something like a third of all published studies fail to replicate in the worst offending fields, and even more fail to replicate at the original effect size or greater. The incentive structure of the entire institution is screwed. Pessimism is warranted here, if you open a journal you should keep in mind that for most fields there’s a double-digit chance that what you’re reading will fail to replicate at all and even more that it’ll fail to replicate at the same of greater effect size.

Given a paper, how do you find out if it was reproduced, disproven, or failed to replicate?

Google “replication crisis” — its huge topic, lots of information.

I think their question was about the replication of a specific method, not scientific replication in general

You can traverse citations to that paper, looking for anything which actually talks about those results.

There are some tools which do this. scite.ai has proven okay for quickly finding some of these citations, sometimes. Nothing beats manually digging in, though.

That and don't just read the abstract or conclusion.

And how would I know if a given paper is of interest for me if I don't read the abstract?

Don't just read the abstract. I didn't say to not read the abstract.

The reason I say this is that there have been numerous cases where someone has tried to discredit one of my claims by citing a paper, and then I actually Sci-hub it and find out that the body of the study only weakly supports the conclusion, or that the study protocol contains a huge caveat not described in the abstract.

I now noticed that I read your comment in the opposite order: "just don't read the abstract". I am glad I asked you, though, because I see your point and I agree with you for I have seen this happening before.

Reply on news.ycombinator.com