The papers that most heavily cite retracted studies | Hacker News Reader

The papers that most heavily cite retracted studies

nature.com

57 pointsXzetaU816 comments

There is a funny kind of bias here: Fields and subfields who actually retract papers are more likely to win the this negative award of "most heavily citing retracted studies" — contrast that psychology, where things are almost never retracted, and there is no way of knowing whether a paper has been replicated consistently except reading hundreds of other papers: no chance of such criticism being leveled.

This is a very good point, survivor biais in all its glory.

I would be interested to see which fields have the most retracted papers, broken by impact index. I am not sure the conclusion to draw from there (most honest vs least competent vs most reproductible vs most scrutinized etc.)

soft “science” vs hard science

Fan favorite of the HN crowd I know, but hardly the correct line to draw.

Here is one of the hundreds of papers analyzing mislabeling in genetics (notice how affected papers are usually not withdrawn): https://link.springer.com/article/10.1007/s00204-015-1632-4

(Unless you consider biology soft)

The article also alludes to this, but doesn’t make it specific:

It doesn’t necessarily say much if a paper cites a paper that has since been retracted.

> Unlike (retracted, 2015) we found …

> While (retracted, 2021) focused on … the present study concerns …

You’d need to do much more in depth analysis to determine to what purpose a retracted study was cited before you can establish whether there is a problem or not.

From dev perspective this highlights that lack of embedded version control and handling of big/security fixes for any academic paper.

Dependencies that are reported to have bugs reported akin to ‘security flaws’, should lead to updates to papers with dependencies.

Naively would be good to sort references into critical references and non-critical references.

It would be good if people would universally use any of the extant tools for citation management. It’s not a problem in CS and associated disciplines but in the humanities and social sciences a shockingly large number of researchers don’t use tools like Zotero and Mendeley. They just keep notes and copy/paste entries in a bibliography.

Can you expand how that would help with this problem? In my understanding the tools used to write a paper are completely hidden and non-consequential once a paper is published, no?

I don't know about the others, but Zotero automatically informs me of retractions for articles in my database: https://www.zotero.org/blog/retracted-item-notifications/

So, if a previous article is retracted when I write my own article, I'd probably know.

I blame rent-seeking academic "publishers" for the lack of these features. There are multiple high-impact features to be built in this regard, many of which would take an experienced dev around an afternoon to build, if a proper data source existed.

Instead, the only way to access many papers is as an idiosyncratic PDF, often only including a post-hoc OCR layer, and of course locked behind a 49.99$ 24-hour-access paywall that can also not be automatically accessed, even if you had the required millions laying around.

Unfortunately, by far the worst offenders in the social studies never retract.

The social studies are well known to have a replication problem, but you don't generally retract a paper just because it can't be replicated. Even if a conclusion turns out to be wrong you generally don't retract it.

Retraction usually connotes some level of fraud or other serious and deliberate problem. It's a serious black mark on your record, which we don't do merely for being wrong. Science is all about hypothesis revision, and we expect to be continuously wrong.

There should perhaps be a designation for "yeah, turns out we were wrong". Instead, you're expected to just find the contradicting papers in your literature search. Saying "I have a paper which proves X" isn't a complete argument unless you've done the full search of relevant papers.

That doesn't fix the replication problem in the social sciences, where the subject (groups of people) are too hard to model and control. People do too often cite a single tiny study as if it meant something, and that's just not a valid way to use science.

But retraction isn't the solution, not so long as the term connotes malfeasance. Like all sciences they need the room to be honestly wrong, because that's fundamental to science.

Maybe instead of just green marks and black marks, we need, uh, brown marks too.

For producing b.s. that doesn't quite rise to the intentional fraud level, but, on balance, detracts from humanity's wisdom and knowledge.

We may call it with some other term than retraction, in order to be clear about the difference, but without "brown marks", there is nothing to stop entshittification of science.

I seriously challenge this mentality / interpretation of a retraction signalling fraud. This is absurd, and actively harmful.

Quite the contrary, a retraction is often a signal of good scientific citizenship and due diligence, which may well come from due diligence of the authors themselves, not just external validators.

NOT retracting a paper out of fear of being labelled a fraud and ostracised from the scientific community is a massive, unhealthy disincentive which we DEFINITELY do not need in the sciences.

The article is VERY careful not to paint retracted papers with such ludicrous stigma; the focus is very appropriately on authors who add "inherited" citations without the necessary due diligence (i.e. actually attempting to read the paper in question, and finding out it has been retracted, for whatever reason)

papers citing retracted studies should be auto-flagged for peer-re-review.

Great idea! The challenges one can imagine with that are:

1) human reviews are expensive

2) the citation might not have anything to do with the reason why the cited paper was retracted.

Now, if it all could be automated (say with some emerging technology that can grok the context of the citation and say classify it into flag/no-flag buckets say), then we are cooking with some real fire. :-)

Reply on news.ycombinator.com