Hacker News Reader

Comment by tlb | Hacker News Reader

Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields.

If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much.

Possible arguments include:

- Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, at least.)

- The 1% is wrong in a more harmful way than the 80%. Perhaps falsifying data is worse than hand-waving conclusions.

So if the maximum upside is 1% of wrong research removed, and the downside is quenching some fraction of the good 19%, it's probably better to leave it alone.

View on HN

The author is arguing for an expansive definition of fraud that includes things like p-hacking. Probably a lot more than 1% of research is p-hacked; that general category of problem might even account for a majority of your 80%.

I'm guessing throwing out "bad data" happens in a lot more than 1% of cases as well.

I think the disproportionate harm argument is a big deal. A single high-profile case of fraud undermines trust in the entire field, and that’s critical to maintain.

I don't think that's true. There have been frauds in materials science, but people still trust it. The fields that are untrusted are the ones that are so hard that it's plausible that the conventional wisdom might be wrong even if everyone is honest.

It's hard to know that a worthless research paper will always be worthless.

Also, who decides what is worthless?

>1% of research is fraud

You are way, waaaaaay underestimating the extent of the problem.

I think you didn't understand the grandparent poster's point. They're arguing, basically, that only 1% of research is deliberate fraud, but 80% is simply useless garbage. The reasons for the garbage are varied, but any way you slice it, it's worthless. So why worry so much about the maliciously useless when there's so much incompetent uselessness?

That matches my experience back when I was an active researcher. Intentional fraud is rare (and highly damaging). Garbage research and garbage papers are everywhere. And even most of the non-garbage research is useless! The core differentiator there is that we just don't know which things will be useful without hindsight. But we can identify plenty of things that were not helpful to anyone and we knew it... yet we still funded those programs.

I think we're mostly in agreement that the size of the problem is vast, even if we might disagree on labelling.

Perhaps I was in an extremely corrupt environment (which eventually made me leave), but I would put the "intentional fraud" rate as high as 30% or so. I pointed out one of such instances once and that was the beginning of the end for me in that place.

More: My field was Biology. No one double checks anyone or anything. A measurement comes back and you can just pretty much ignore it, in the end you just write whatever you need for your thesis to hold (I mean not me, but that's pretty much everybody's dirty secret). In the rare case that someone actually wants to check it out (1 in 1,000), you can just say that the "original data was lost". Sometimes the only thing you can trace back is some sort of written journal, in which anybody could write anything and that doesn't make it true/false.

Not long ago, there was a group of scientists that went on to try to replicate some landmark results on cancer research (I'll update with a good reference if I find it) and found out that only a very low number of them matched their observations. Some of those were not even close ... like, waaaay outside a very generous experimental margin of error.

Ouch. Things certainly do vary from field to field and I was in one of the better ones (particle physics), so I believe you.

Given how hard an environment it is even in the honest research groups, that must have been awful.

> I pointed out one of such instances once and that was the beginning of the end for me in that place.

How does that occur? Did they invent some reason you couldn't remain? Were you intending to remain anonymous when reported it? Lastly, were there consequences for the fraudsters?

Power games.

In my specific scenario, she said I was underperforming and didn't want me anymore as her student, there's nothing much one could do after that.

Of course I tried to defend my stance, even provided solid evidence of her wrongdoings and overall attitude towards me and other students. In the end it was "just better if you just leave".

>Were there consequences for the fraudsters?

Zero.

Reply on news.ycombinator.com