Defence against scientific fraud: a proposal for a new MSc course
deevybee.blogspot.com
deevybee.blogspot.com
My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common. Researchers invest a huge amount of effort into a study, they really want the hypothesis to be true. A lot of studies have steps that are pretty susceptible to bias, like maybe you’re classifying results and there’s a bit of a judgement call there. It’s human nature to make biased decisions in these situations, and make enough of these biased decisions, and statistically insignificant results become statistically significant.
Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive. While I see bias as more, you’ve got a borderline case, and you view it in the light you want to see it in, even somewhat unconsciously. Like the difference between unconscious racial bias, and overt hateful racism.
I think the best approach to combatting this is to spend a lot less of the overall $$ in science on novel research, and a lot more on attempting to independently reproduce results. Papers should be seen as meaningless until their results can be independently reproduced, and universities/colleges should reward reproduction studies as much as novel research. It’s kind of crazy that the system almost completely lacks these checks and balances right now - peer review is more like an editor, it’s just a very different thing than reproduction.
Bishop here is suggesting a data sleuthing approach to root out fraudsters, but I dunno if that’d be effective, as I think the main issue is subtle but pervasive bias seeping into studies by most researchers, vs. a smaller number of heavy fraudsters. Independent reproduction of results, while expensive, is the only effective approach I can think of to combat this.
There's also a grey area between fraud and not checking your work as thoroughly as you should.
For instance, I've witnessed a highly regarded researcher (Turing award) telling his co-author not to bother about some proof because nobody would read it.
I have a similar experience and opinion (started a PhD, quit when I lost faith in the institution).
In our field in particular, it seemed to be an open secret that our current paradigm was a dead end, and this had been apparent for almost a decade by the time I joined the university.
Yet, in spite of this fact that we were going nowhere, there was extreme pressure to continue to toe the line. You couldn't just try something new based on a hunch, but instead had to perform fruitless experiments that were "guided by the literature" and therefore easier to justify to the people funding your endeavour.
The whole institution of academia is rotten, and going on a witch hunt like the article suggests ignores several massive elephants in the room.
Lets say you do three runs of the same psychology experiment and only publish the results from the one where results are most significant. I think most researchers would not consider this fraud. You are publishing data that's absolutely true. It's the selective cherry picking of data in order to produce novel results that's rotting the field.
This is how I think researchers can sleep sound at night while faith in science continues to erode.
That's not what happened, and they discredited themselves. Guess what. The public does know what the scientific method is, and they smelled a rat.
Why should we trust scientists if they violate their own principles?
> continuous attacks
Yeah. These are often warranted. Scientists MUST be held to a higher standard because (unless they're funded by HHMI) they're using the public purse.
> diminishing returns
That's a real thing. It's not supposed.
As someone working at a university, it's definitely not the dominant ideology amongst the decision makers. Maybe in some humanities departments, but not in the rest of the university, and especially not in the administration.
Science had a religious like following, doctors were considered infallible, and so on
The fact that it is a well established notion that pharmaceutical companies buy studies is a new phenomena (relatively to before post modernism set in)
The extreme manifestations is anti-vax, and it’s usually not due to them coming out of a humanities department
If science wants to wear the crown of supreme human cognition, I really wish they'd educate their fan base about how it should be done.
Maybe that’s not a good example, I’m not a psychologist. But in my experience, scientific studies are full of a surprising number of judgement calls, in all sorts of areas (experimental design, experimental execution, result classification, statistical analysis, exclusion of outliers, etc.), and it’s naive to think bias doesn’t seep into all of these, even for people who are quite committed to the ideals of science. I can’t think of a good way to combat this except very extensive independent reproduction of results.
— Richard Feynman
Then, in order to not fool yourself, you have to know your own weaknesses:
<https://en.wikipedia.org/w/index.php?title=List_of_cognitive...>
BTW scientists get no upvotes for merely reproducing "known" results.
Is there a way to see natural merit/discovery in reproducing (not an artificial incentive, like grants for repeating studies). Perhaps analogous to how teaching helps your own understanding?
The problem here is perhaps one of reporting negative results.
It'd be very easy to rationalize that a secondary study didn't work because, hey it's not direct reproduction and this use case is invalid or the equipment available wasn't the same and etc...). And between the rationalizations and the lack of incentive to submit and publish negative results it's easy to extrapolate that an indirect challenge may take a really long time to be put forward.
Something kin to a bug report section might be in order. A "Works on my machine." regime isn't acceptable in the empirical domain. This would help prune dead ends in both primary and secondary+ research, one would hope, without necessitating direct redundancy. Of course this may run against the grain, fears of getting scooped or IP challenges and etc...
The experiment shouldn't inherently be designed to confirm your biases. I've carried that with me ever since. I agree with you - I suspect bias, judgement calls that favor what you want, looking for errors intensively when you get weird results and not when you get expected ones, etc. are far more common than outright fraud.
The problem is that peer review right now only provides the appearance of oversight. Referees are not given the time, money, or resources to really dig into papers and look for issues. I have peer reviewed 4 papers this year, averaging around 3 hours per review. The onus is largely on the referees/editors/publishers to immediately find something wrong in the paper to reject it rather than on the authors to really prove their case. And I mean "immediately". One journal that I review for requires peer reviews within 2 weeks of accepting the assignment, and they will nag you if you do not have it done within a week. If the onus must be on the referees, they need to be given much more time and resources, and that might even include money, and also might include waiting for reproduction of the results before publishing.
"Publish or perish" also plays a big role here, because scientists need to get papers published to prove they are productive and worthy of funding and employment. Authors will just re-submit their manuscripts to different journals until they are published rather than re-evaluating the research in any fundamental manner (is my methodology flawed? etc.). So putting the onus on referees isn't going to work in the first place, since peer reviews are not always shared.
This is like asking merchants to catch and stop fraud and crime (e.g. selling alcohol to kids); it's in their best interest to not catch fraud and maximize their income.
The reason they do clamp down on underage drinking is because they'll get fined/arrested / their license will be revoked. The system cares enough to catch and punish the behavior.
If the funding bodies like NIH don't catch, punish and stop this behavior, it creates a system where fraudsters win more. This makes more groups, even if reluctant, participate in fraud because that's the only way to compete. It's a race to the bottom.
Money drives incentives, and clawing it back while blacklisting and publicly humiliating a lab, will change behavior. That would make the lab a toxic collaborator, especially if collaborators ALSO get blacklisted from funding.
But this is the same system that is unable to move off paid journals when the web has allowed ~0 cost publishing for decades. They can't effectively retract papers when the knowledge has failed to hold up. Like we have so many ways to organize, publish and review information on the web but they haven't adopted any of them.
Literally a clone of HN or Reddit with upvotes and comments could solve half the problems with publishing and peer review. Actually publishing your data and your scripts no matter how hacky they are can help replication. This is a problem that is technologically easy to at least attack and improve if not solve. But none of that has happened so i don't have much hope, for some reason they aren't able to change.
Some fields, like machine learning, gets deluged with papers coming from all kinds of researchers - there's no unified scientific notation, so to speak. In one paper you could have a pure math authors, in some other you could have econ authors, and then maybe a theoretical physics author. All these could use wildly different notation to describe related ideas, or discuss ideas to solve problems in their respective fields.
And in that case, how do you detect bogus science? You could be a leading ML scientist, but not have the faintest idea about the other domain. Or the inverse - be a leading domain expert, but have little ML knowledge.
Not to mention, the people that are publishing fraudulent scientific papers aren't morons - they're likely publishing that kind of stuff because of bad incentives. Maybe you'll catch the laziest, least capable fraudsters - but trying to catch savvy researchers sounds like a daunting task.
In my mind, this type of task/job is more fit for full-time scientific "auditors".
There is no incentive mechanism, it has been the problem all along. The peer review process is clearly inadequate or not up to the task in its current form.
(via https://news.ycombinator.com/item?id=38336432, but we merged that thread hither)
https://news.ycombinator.com/item?id=28107614 ("Tortured phrases: A dubious writing style emerging in science"—256 comments)
> ... sideline any honest young scientists who want to do things properly. I fear in some institutions this has already happened.
She has some names in mind!
https://www.npr.org/2023/07/19/1188828810/stanford-universit...
> The president of Stanford University has resigned after an investigation opened by the board of trustees found several academic reports he authored contained manipulated data.
> Marc Tessier-Lavigne, who has spent seven years as president, authored 12 reports that contained falsified information, including lab panels that had been stitched together, panel backgrounds that were digitally altered and blot results taken from other research papers.