It’s sad that as an electorate, as a people, (even the democrats), show a lowering of “confidence” in higher ed. It means experts and scientists get ridiculed. It’s disgusting and sad.
Is this a harbinger of a failing empire? I think so.
Given the shameless plague of fraudulent papers, and the gaming of citations, my faith in experts and scientists has been damaged.
Then there was the "trust the science" fiasco around covid.
Trust is like glass - easy to break, and very hard to restore.
doesn't mean there isn't a lot of local fraud. Elizabeth Holmes comes to mind.
Let's not forget that fraudulent Alzheimer's paper that misdirected Alzheimer's research for decades.
The Pons and Fleischmann cold fusion fraud?
Does it matter if most come from foreign scientists, when prestigious journals publish them? What are we supposed to do, reject papers with foreign sounding names? Why aren't the journals doing proper vetting?
Attempts to suppress any counterclaims to the official scientific opinions during covid would make anyone skeptical.
Then there's this:
https://www.oaklandreport.org/p/the-misrepresentation-of-oak...
https://www.wsj.com/articles/the-media-say-crime-is-going-do...
One should be careful about statistics reported by an entity that has a strong self-interest in the outcomes.
* A single horrible murder took place decades ago * A second (possibly less terrible) manslaughter case that took place in the 1980s * A variety of violent acts in dangerous areas in developing countries, today
I’m not even going to dispute that there is academic fraud. Just that your examples tend to illustrate how relatively rare it is when one filters for reputable researchers, journals and institutions, and also takes into account the denominator (which is enormous) rather than highlighting some spectacular examples and making broad claims about the world falling apart.
I've also seen articles (didn't keep cites) about the "replication crisis" of large numbers of research papers with results that nobody can replicate.
> making broad claims about the world falling apart
More like claims of losing trust in "trust us, because we're scientists!"
If there isn't good evidence to show that most science is good, the best we can probably claim is that we don't know.
With respect to your question: “is there evidence that most science is good”, I’m not even sure how you would answer that question. All I know is that if you compare the world of my grandmother to the world of today, the two are so different as to be unrecognizable, and scientific advances are responsible for most of the difference. That progress continues every single day, and we’re both alive because of science was at least good enough to make our continued existence possible.
>I’m not even sure how you would answer the question.
It should probably be answered in the same way we’d answer the converse. Something like “here is the data and the tests that show the data quality is high and the results are replicable.” My point was, if you can’t show that, then the claims that the work is good may not be scientifically valid.
To put a finer point on it, we know there is replicability crisis in many corners of science. But we don’t likely know how much of that is due to bad science or outright fraud unless we do the science-y work to check. And unfortunately many of the incentives are aligned against that very thing.
The post I responded to said: “given the shameless plague of fraudulent papers, and the gaming of citations, my faith in experts and scientists has been damaged.” I picked a bone with that given the total lack of evidence for the claim. More generally I have a problem with a broader class of people who make claims like this, since I find these claims unlikely and unjustified.
You came back to me with a different question: “how do we know that all science is good?” I genuinely don’t know how to answer that question. And bluntly, I think it would be incredibly hard to answer it: the question is up there with “how many grains of sand are there on the world’s beaches.” I do think that clearly science, in whatever state it is in, has historically worked amazingly well. The world we live in is a testament to that. Maybe the answer is that it doesn’t matter, that ecience has always been mediocre and yet we go to space and cure diseases with even low-quality science.
A more general answer is that everyone doing science is working incredibly hard to find the answers to hard questions. We are always doing so imperfectly, with imperfect methods and imperfect tools and imperfect brains and yet we soldier on. If you want to join this effort, either to further it or bring evidence to criticize and improve it, we are all excited to have you. But simply posing impossible-to-answer questions is not necessarily doing either thing.
I think this is too romantic of a view. Yes, some (or even many) are working hard to answer hard questions. Others are seeking money, status and/or prestige as their top priority, just like practically any other field. It’s no different than saying financiers are working incredibly hard to create a rising tide that lifts all boats. It makes for a lofty narrative, but it’s also ignoring much of human nature.
And if you think that question is impossible to answer, it should at the very least give you pause to think about the confidence you have in your conclusion. Otherwise it reads as “I can’t prove it, but I know it’s true.” That’s dogma, not science.
But so what? What’s most interesting about the conversation we’re having is that even as folks make and fall back from specific arguments, the directional nature of the discussion doesn’t shift. All this tells me is that you don’t like scientists and are eager to land a punch against them, even if the punch is as weak as “they’re human.” This kind of conversation doesn’t have much value to me, since I can’t learn anything from it except that you have some strong emotions. But maybe it could be valuable to you?
I think you’re extrapolating too much. Again, you’re making a strong claim based on limited evidence which is the hallmark of bad reasoning. I work in research and publish/referee articles regularly. I have no qualms with scientists, but I’m also acutely aware of how the system is flawed.
>I think the first claim is made without evidence
I get the impression you’re too riled to actually see my point so I’ll be explicit. We have lots of evidence that bad science is pretty widespread, especially in certain disciplines. We also have anecdotal examples of fraud. The latter is a subset of the former, with the remainder being good work. How big of a subset is it? Your claim is that it's small; my claim is we don't know.
There’s no goalpost shifting. All of the points made are related to the issue of fraud. Asking for evidence of your claim is not goalpost shifting, nor is understanding that fraud is a consequence of human incentives for status that exist in practically every domain. I simply made the point that we know there’s a problem, but likely don’t know the extent of it. Meanwhile, you made the claim that academic fraud is rare. And yet you also claim it’s impossible to prove that it’s not rare. It’s pretty clear that you’re not making a reasoned argument, which is the irony of someone defending the current state of science.
I read back through my posts to find the claim I made that you’re talking about and I can’t find it. Please let me know what you’re talking about, maybe quote it. I’m gathering this has mostly become a troll at this point, and I’m sorry I kept responding.
>examples tend to illustrate how relatively rare it is
Given the fraud and replicability issues that have been shown to exist even within the most reputable institutions and journals, I think this claim needs some substance to back it up.
>this has mostly become a troll at this point
Please review the HN guidelines. They specifically reference assuming good faith and avoiding shallow dismissals. I’ve tried giving that to you by asking questions and giving you multiple opportunities to give evidence to your position.
1. The initial post made a strong claim about high levels of fraud. 2. I asked for evidence to support this. The evidence given was arguably very sparse.
The quote you shared is from my response to point (2). Arguably in a world of infinite pedantry it could have been phrased differently. For example: “given how relatively rate cases of fraud supported by actual evidence are” might have been clearer. However given the context I assumed that claims of scientific fraud would involve evidence.
This seemed a good place to end the conversation. Yet:
3. You came in and “judo flipped” the conversation to “aha; but what about the fraud that could theoretically exist but that somehow had produced no evidence of its existence. Perhaps these levels are very high!” 4. I said that evidence of absence would be enormously difficult to prove, given that it would require resources beyond those we currently spend on science to identify this. 5. And moreover I pointed out that we have never had the means to detect this evidence-non-producing fraud and yet science has managed to produce massive and visible benefits that we experience on a daily basis.
And thus this leaves us at an impasse. All science may be full of fraud and bad results that we cannot detect. Perhaps it always had been! And yet this has not prevented science from producing results that are enormously valuable. I am left with some questions:
1. What strategy would you pursue to answer your extremely difficult question, given that real-world resources are extremely finite. Please do not say “well, we just need to replicate every result” becayse this simply misunderstands the point about resources. 2. If we cannot devise a feasible policy to test your “perhaps science is full of fraud” question, what should we do instead? Is there an actionable policy you’d pursue?
My view is that strong answers to both questions are not so much interesting to me, but interesting in terms of whether your world-view has value. Surprise me!
The other commenter did provide some evidence (albeit anectodal) of obvious fraud. In the context of non-replicable studies, it can be as high as 2/3rds are shown to be non-reproducable in some domains. This includes some of the highest quality journals and institutions. There's quite a bit of evidence of this. Now I'm not claiming that is all fraud, but I am claiming we don't know how much of the problem is fraud.
>evidence of absence would be enormously difficult to prove
I disagree. I'm taking the point that it's not hard to prove, but rather the incentives are aligned against it. If data and methods are truly open-sourced, much of this can be proven. But there is little status or money to be gained from reproducing work, leading to a situation where we have a system where it is rarely done.
>yet science has managed to produce massive and visible benefits
I don't disagree, but I think you're are misinterpreting my position. I said the system is flawed, not that it's worthless. Put differently, it still produces good science but it's hard to separate the good science from the bad. The signal-to-noise ratio is potentially low, and the rate of publication exacerbates this. I would argue that boosting the signal-to-noise ratio by weeding out bad science could produce much more benefit because less time is wasted chasing trails of bad science.
>What strategy would you pursue
You keep framing this as some impossible scenario. I'm saying it only appears that way within the context of the current incentive struture. So my goal would be to create a change to that incentive structure. This is off the top of my head, so don't take it too seriously but here's what I would try:
1) We need to change the way academics are measured. Currently, it's a "publish or perish" paradigm that puts a premium on novelty and essentially disregards most everything else. There is also a glut of PhDs; far too many given the number of tenured academic position. This puts an enormous amount of pressure on a PhD to produce publications and this can manifest itself in pushing bad science. There's two things we could do to help this:
1.(a) Provide an incentive where reproducing research is valuable. Currently, researchers don't want to spend time reproducing studies because there's very little upside. They won't generally get published so it doesn't help their career. I would say we need to start giving more status to reproduction studies. Probably not as much as more "novel" studies, but there should be an avenue to publish them. Maybe a two-tiered journal structure where we have *Nature* and *Nature:reproductions*, where the latter is used to publish works that re-create the studies in the first. Metrics like citeScore could be weighted so that the novel studies give you a higher metric than the reproductions, but you still get some credit for either.
1.(b) Provide a pathway to also publish "non surprising" studies. In the current structure, research is more prioritized if it produces surprising results. Thus, if a researcher produces work that is unsurprising, they tend to just throw it out and try something else. This can lead to wasted effort when another researcher ends up doing the same thing. We should create an avenue to also publish "unsurprising" results. This has already started to get applied in some domains, like economics.
2. Right now, publications can hide their data. I think there should be a cultural shift where the highest quality journals make sharing the data and code a mandatory requirement. This ties directly in to facilitating the replicability as well as allowing others to look at your methods and see if there's any wonkiness going on (whether fraud or otherwise). Some of the issues of fraud that we've seen are because researchers couldn't hide their data because it came from a third party (eg, Dan Ariely and Francesca Gino were outted because their data was owned by an insurance company that was willing provide insight into the raw data).
3. Journals should set aside a portion of fees to fund replication. As it stands, journals get free labor from referees. I would argue this should be balanced by having some money set aside to fund a process aimed at replicating some of the studies in journal. It's wild to me that we have a system where a referee is expected to work for free and also to pay for a subscription to the journal they are helping.
Like Fauci with the masks. He knew he was wrong.
I'm old enough to remember when the government confidently asserted that butter was bad, eat margarine instead. I ate margarine for decades. Then it turned out the hydrogenated oil is more or less poison. Back to butter for me. We were told eggs were bad, meat was bad, grain is good. All wrong.
Also I'm surprised the academic integrity crisis hasn't been mentioned yet... The cheating and plagiarism rates are orders of magnitude worse now than pre pandemic and LLMs
You forgot lobbying.
Are scientists working for the government free from incentives to push bad science?
Of course the problem with it all is one general of medicine. The infectious disease specialist stays in their lane and devises an infectious disease solution. I don't think psychiatrists or economists had much of a say in it. I would speculate there are particular populations such as healthy teenagers, who mostly handle COVID easily with near-zero death rate( HEALTHY teenagers).. where these COVID restrictions caused more damage, psychological and suicide, then the COVID precautions themselves... in order to save the boomer population which did benefit.
The other problem, obvious now, is it didn't work. You can't bottle up people, particularly not muricans, a very individualistic culture. The goal was to knock out COVID in a week or two and look how it went.