Hacker News Reader

> Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive.

Lets say you do three runs of the same psychology experiment and only publish the results from the one where results are most significant. I think most researchers would not consider this fraud. You are publishing data that's absolutely true. It's the selective cherry picking of data in order to produce novel results that's rotting the field.

This is how I think researchers can sleep sound at night while faith in science continues to erode.

Erosion of faith in science (whatever that means) is not due to research fraud lol.

whats it due to then in your opinion?

There are many reasons, including continuous attacks by politicians on scientists and the scientific endeavour, inaccessibility of modern research to the average person, (seemingly) diminishing returns from scientific research, to name just a few.

Do you think there is anything wrong within the scientific community, or do you think the diminishing trust is completely caused by outside forces?

I think the scientific community is complicit in some of the issues I listed, but they're not 100% responsible.

Well scientists are shitty. For example, the entire masking bullshit could have been avoided if the science policymakers just said "hey we don't have any evidence, just dont be an asshole and put a mask on".

That's not what happened, and they discredited themselves. Guess what. The public does know what the scientific method is, and they smelled a rat.

Why should we trust scientists if they violate their own principles?

> continuous attacks

Yeah. These are often warranted. Scientists MUST be held to a higher standard because (unless they're funded by HHMI) they're using the public purse.

> diminishing returns

That's a real thing. It's not supposed.

Poor science journalism. The joke that depending on the day of the week coffee is 'scientifically' either the best or worst thing in the world is what's killing trust in science. Most people aren't reading or hearing what scientists say directly, they're hearing cut up overly summarized tidbits or reading a clickbaiting journalist's interpretation of a paper that most of the time the journalists don't even consider to be worth linking to.

Notably, public university press departments are scarcely any better in this regard than for-profit popular science publications. These are journalists who should have direct access to the researchers and they still get stuff laughably wrong.

Post modernism. There’s a well established field of thought going past to at least the 60s that believes truth is relative, knowledge is subservient to language and therefore science is inherently biased and a tool of “power”. It’s extremely popular in the west, especially among the elite and in the universities

Post modernism is a relatively niche ideology.

As someone working at a university, it's definitely not the dominant ideology amongst the decision makers. Maybe in some humanities departments, but not in the rest of the university, and especially not in the administration.

I think you are underestimating the effect.

Science had a religious like following, doctors were considered infallible, and so on

The fact that it is a well established notion that pharmaceutical companies buy studies is a new phenomena (relatively to before post modernism set in)

The extreme manifestations is anti-vax, and it’s usually not due to them coming out of a humanities department

It's not possible for you to objectively know the extent of erosion of faith in science or the causes, you can only speculate.

If science wants to wear the crown of supreme human cognition, I really wish they'd educate their fan base about how it should be done.

That I would consider fraud, personally. I’m referring to more subtle bias than that - like maybe you’re putting someone through a scenario, then trying to classify their emotional state afterwards. You classify the clear states properly, but are biased in classifying the borderline ones - like it’s not very clear if they’re sad or anxious, but if anxious fits your hypothesis better, you tend to classify these unclear states as anxious.

Maybe that’s not a good example, I’m not a psychologist. But in my experience, scientific studies are full of a surprising number of judgement calls, in all sorts of areas (experimental design, experimental execution, result classification, statistical analysis, exclusion of outliers, etc.), and it’s naive to think bias doesn’t seep into all of these, even for people who are quite committed to the ideals of science. I can’t think of a good way to combat this except very extensive independent reproduction of results.

The situations you're describing should be prevented by a double blind trial - they're actually exactly what a double blind is meant to prevent.

You're actually thinking of a triple blind trial: i.e. blinding the experimental subject, the scientists administering the experiment, and the statisticians interpreting the results. Double-blind is only the first two.

Right, thank you.

Yeah, I never worked in a field that did double blind studies - did my MSc in Oceanography, and these were areas I felt bias could (and likely generally did) seep in within Oceanography. I assume each field has its own weak points, but I’d assume most fields do have them, since there’s so little real, independent reproducing of results, we’re basically depending on individual scientists to be perfectly objective, which just isn’t how human beings work IMO.

Interestingly, this correlates with fraud in journalism as well. One common technique is that there's a set of events that could be written about / published, but they choose to only write about the events that positively promotes their ideology, while the events that might negatively impact their ideology are ignored and not shared to the world. It's one of the ways that a news organization can "lie" without ever actually telling a lie. By painting a picture of the world using selectively chosen data based on a bias/agenda, rather than a picture of the world that is based on all available data.

Someone should compile a master list of various forms of humans deceiving each other.

“The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.”

— Richard Feynman

Then, in order to not fool yourself, you have to know your own weaknesses:

<https://en.wikipedia.org/w/index.php?title=List_of_cognitive...>

Given we teach that in our biomedical ethics course as fraud, I think your "most researchers..." statement is not necessarily supportable.

> My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common

There's also a grey area between fraud and not checking your work as thoroughly as you should.

For instance, I've witnessed a highly regarded researcher (Turing award) telling his co-author not to bother about some proof because nobody would read it.

Yeah totally, good example of the somewhat subtle ways bias seeps in.

Feynman has a bit where empirixal measurements of some universal constant (electron charge?) slowly crept up to the now-accepted true value. He said scientists are a little bit ashamed of this, because it showed that higher estimates were discarded and not published - bias.

BTW scientists get no upvotes for merely reproducing "known" results.

Is there a way to see natural merit/discovery in reproducing (not an artificial incentive, like grants for repeating studies). Perhaps analogous to how teaching helps your own understanding?

https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's...

>My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common.

I have a similar experience and opinion (started a PhD, quit when I lost faith in the institution).

In our field in particular, it seemed to be an open secret that our current paradigm was a dead end, and this had been apparent for almost a decade by the time I joined the university.

Yet, in spite of this fact that we were going nowhere, there was extreme pressure to continue to toe the line. You couldn't just try something new based on a hunch, but instead had to perform fruitless experiments that were "guided by the literature" and therefore easier to justify to the people funding your endeavour.

The whole institution of academia is rotten, and going on a witch hunt like the article suggests ignores several massive elephants in the room.

Replication is kind of fuzzy though. A lot of things are replicated, just indirectly, building on suppositions supported by but not directly reproducing the study.

The problem here is perhaps one of reporting negative results.

It'd be very easy to rationalize that a secondary study didn't work because, hey it's not direct reproduction and this use case is invalid or the equipment available wasn't the same and etc...). And between the rationalizations and the lack of incentive to submit and publish negative results it's easy to extrapolate that an indirect challenge may take a really long time to be put forward.

Something kin to a bug report section might be in order. A "Works on my machine." regime isn't acceptable in the empirical domain. This would help prune dead ends in both primary and secondary+ research, one would hope, without necessitating direct redundancy. Of course this may run against the grain, fears of getting scooped or IP challenges and etc...

One of the first things a professor taught me on a class on the design of experiments is "Never doom yourself to success."

The experiment shouldn't inherently be designed to confirm your biases. I've carried that with me ever since. I agree with you - I suspect bias, judgement calls that favor what you want, looking for errors intensively when you get weird results and not when you get expected ones, etc. are far more common than outright fraud.

Starting point of the article appears to be that this gut feeling you mention, that there is very little outright fraud, is not completely right. It appears to me that he claims that there is more outright fraud than we believe.

As a sociologist put it, it is human nature to mislead other humans, sometimes knowingly.