Hacker News Reader

Comment by Palomides | Hacker News Reader

DARPA is doing a relatively large scale effort to see if social sciences papers replicate, and how easy it is to look at a paper and guess if it will replicate

the preliminary estimation phase predicts like 50% rates of replication

let's be real, the average social science paper is not good

edit: there are lots of garbage papers in every field, the culture of academia is deeply flawed, the low standards are just even lower in some fields

View on HN

The problem is that replication studies aren't done hard enough.

And by replication studies I mean "take this paper, try to replicate, write a paper on it".

Every singular paper can be wrong, that's why you're supposed to replicate them - that's at least what was taught in psychology, which many consider "social science of less rigour" - well, it was probably the course most hardcore about experimental design in my education and ability to replicate a study...

Are they not done or are they done and nobody wants to publish them?

Nobody CAN publish them. An actual criterion for most top journals is literally "novelty". Reviewers will immediately levee "this has been done before" as a damning critique in peer review. There is no replication because you can't currently share replications, let alone sustain a career on them.

My experience of "representative" studies in psychology (university of Berne) was that they'd come up with something, then gather a group of "volunteer" psychology students - 80% women - who need to participate at x studies as requirement to pass, publish it as representative serious research, and call it a day.

>The problem is that replication studies aren't done hard enough.

>And by replication studies I mean "take this paper, try to replicate, write a paper on it".

You don't necessarily need to perform a straight-up "replication study" in order to do some useful sanity checking on published results. It can be done as part of another novel study that builds upon prior work. That's basically what happened with the fraudulent UCLA political persuasion study several years ago [1]. Some Berkeley researchers wanted to extend the ideas in that earlier published paper, reused similar methods under the assumption that the UCLA folks were correct, and couldn't get anything reasonable within the ballpark of what was published.

[1] https://fivethirtyeight.com/features/how-two-grad-students-u...

Yes there’s a serious issue, but thrust of the previous comment was that it’s not productive to criticize poor research using poor research.

Conservatively 90%? Bullshit. The real numbers are already terrible so why use unintentional hyperbole to make the argument it just weakens it.

I don't think there's any objective way to say that a majority of papers are bullshit, even if they are. Failure to replicate is "easy" to show, but doesn't say anything about the utility or quality of the science done, beyond the absolute basics of a paper describing what a scientist did in detail and not lying about the results. A "sky is blue" paper will replicate, but I'm not sure how you'd find a "fair" way to call it useless.

Anyway, jumping from "50% of papers will probably fail an absolute bare minimum test" to "90% of papers are garbage" is subjectively reasonable to me.

If 90% seems shocking to someone, I suggest they skim a bunch of papers and develop their own feel for how bad the situation is.

I read a lot of history and some linguistics, it seems really fine to me. Linguistic especially, they work on _a lot_ of data, you don't have the "tested on 15 subjects" from biology or medecine effect.

Even if i agreed with your "subjectively reasonable" idea, the issue is that non-STEM science encapsulate a lot of different fields.

what's non-STEM science? the incentives of publishing and funding systems are similar across most fields (publish papers in the cool journals, get as many citations as possible, leverage that for more funding), I would be astonished if there were fields substantially exempt from bad science.

I feel like my position is fair for any field publishing papers with falsifiable claims based on evidence?

Possibly you and the OP would be better served to say “I wouldn’t be surprised if some claim about x%” is true. Or, “I would guess it’s true that”.

Back to the main subject Vertasium has a related video:

Is Most Published Research Wrong https://youtu.be/42QuXLucH3Q?si=p9M0ZOAdHuowD30n

Reply on news.ycombinator.com