Star neuroscientist may have manipulated data to support a major stroke trial
science.org
science.org
> Under Zlokovic’s leadership, the USC institute has expanded to more than 30 labs and grown its annual funding more than 10-fold, exceeding $39 million in 2022. NIH grants to Zlokovic have totaled about $93 million. A prodigious fundraiser, in the past decade alone he has added at least $28 million from private sources, according to USC.
So the NIH has given the man $93m. Surely that's a place to start?
If you establish this, that's the case right there.
https://www.businessinsider.com/reinhart-and-rogoff-admit-ex...
https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
Not-so-fun fact: Economists don't publish their actual Excel spreadsheets, which would have made their error trivial to prove, but just the results!
https://inthesetimes.com/article/the-excel-error-heard-round...
it seems like its basically "If you get money from lying electronically, go to jail"
It remained to be seen if he were actually guilty.
> Carmen Ortiz, the federal prosecutor who hounded Aaron Swartz in the months before his Friday suicide, has released a statement arguing that "this office’s conduct was appropriate in bringing and handling this case." She says that she recognized that Swartz's crimes were not serious, and as a result she sought "an appropriate sentence that matched the alleged conduct – a sentence that we would recommend to the judge of six months in a low security setting."
> That's funny because the press release her office released in 2011 says that Swartz "faces up to 35 years in prison, to be followed by three years of supervised release, restitution, forfeiture and a fine of up to $1 million." And she apparently didn't think even that was enough, because last year her office piled on even more charges, for a theoretical maximum of more than 50 years in jail.
https://www.forbes.com/sites/timothylee/2013/01/17/aaron-swa...
> Some have blithely said Aaron should just have taken a deal. This is callous. There was great practical risk to Aaron from pleading to any felony. Felons have trouble getting jobs, aren't allowed to vote (though that right may be restored) and cannot own firearms (though Aaron wasn't the type for that, anyway). More particularly, the court is not constrained to sentence as the government suggests. Rather, the probation department drafts an advisory sentencing report recommending a sentence based on the guidelines. The judge tends to rely heavily on that "neutral" report in sentencing. If Aaron pleaded to a misdemeanor, his potential sentence would be capped at one year, regardless of his guidelines calculation. However, if he plead guilty to a felony, he could have been sentenced to as many as 5 years, despite the government's agreement not to argue for more. Each additional conviction would increase the cap by 5 years, though the guidelines calculation would remain the same. No wonder he didn't want to plead to 13 felonies. Also, Aaron would have had to swear under oath that he committed a crime, something he did not actually believe.
https://cyberlaw.stanford.edu/blog/2013/01/towards-learning-...
The intimidation of the stacked charges led to Swartz's suicide.
The gov. was 100% responsible for his death, which is why I say they killed him, indirectly, but they still caused the anguish that made him feel like suicide was the only way out.
Sadly this strategy is used all too often on citizens, plea deal offered to say "look they pleaded guilty, we were right", otherwise you get the stick, life in jail.
Not if you believe in personal accountability. He made a choice, he faced the consequences; he made a choice again, he faced those consequences. If I kill a man and am given a very long sentence, then kill myself to avoid the wait, it's not the government's fault I died.
Where is the personal accountability for the prosecutor?
There is a notion of reckless disregard for human life in the law. If you are throwing bricks off an overpass just for fun, you are still guilty of murder if you kill someone - because you have reckless disregard for human life even though you weren't trying to kill anyone.
Likewise, a prosecutor who threatens someone with decades in prison for freely distributing academic material displays a reckless disregard for human life. Even if the prosecutor didn't intend to provoke a suicide the prosecutor was doing something bad with a reckless disregard for human life and it got someone killed. That's murder, in my view.
Perhaps you should ask for accountability from the prosecutor/murderer rather than from her victim.
And he is accountable for his actions of refusing to do that, and furthermore refusing to participate in the matter whatsoever.
Right, so he could have picked the option where he walked free in six months, but instead he picked the option where he's dead.
> Felons have trouble getting jobs
> aren't allowed to vote
> and cannot own firearms
He's not doing any of those now, so I do not see how his would-be restrictions are relevant. The following speculation is just that, and also irrelevant.
> Aaron would have had to swear under oath that he committed a crime, something he did not actually believe.
Ah yes, the "Sure I secretly entered the networking cabinet of an institution I have no formal relationship with, to install my personal equipment into it without their knowledge (much less consent) in order to exploit their access levels to fulfill personal objectives that would otherwise be impossible to me, then hopped around IP's as the institution I was stealing documents from blocked the IP I was stealing until the entire IP range of the institution whose access I was hijacking was blocked due to my hacking and hence all the researchers doing legitimate work at that institution could not access the resources they paid for, but how could I have known that was unlawful?? I fundamentally cannot accept any blame for my actions." defense. I don't buy it.
You missed this part:
> However, if he plead guilty to a felony, he could have been sentenced to as many as 5 years, despite the government's agreement not to argue for more. Each additional conviction would increase the cap by 5 years, though the guidelines calculation would remain the same. No wonder he didn't want to plead to 13 felonies.
There was no guarantee it would only be a six-month sentence.
I'm actually in the "personal responsibility" camp myself, but the amount of overcharging they did to him was obscene. He wasn't facing the consequences of his actions-- he was about to get fucked by the Statutory Ape.
To put it in perspective...Ghislaine Maxwell got 20 years for being an accomplice to child sex trafficking. This is the same sentence given to lesser spies. Swartz committed trespassing, a bunch of copyright infringement, and caused a DoS? It's not cool, but it's not 2.5x worse than pimping children.
If he was willing to admit his trespassing, bunch of copyright infringement, and DoS was wrong he'd in all likelihood walk free within a year. But for whatever reason he was not able to do that, and here we are.
Personally, I'm glad we are in the world where trespassing into a property to hijack their network and cause outages of global services is not considered "minor". Obviously the situation as it played out here is an absolute tragedy, but the matter was indeed serious and was handled with an appropriate level of gravity. All the prosecution was looking for was a "my b, that was wrong" to prove their point - and he refused to give it.
As they say: if you can't do the time, don't do the crime.
No, it was not a 35-50 year sentence. That's the theoretical sentence one could get for those same crimes if all the sentence enhancing factors that can apply do apply. Repeat offender, part of organized crime, massive monetary damage, drugs involved, things like that. Swartz didn't have any of those factors.
Here's an article on how DoJ press releases ridiculously exaggerate potential sentences [1].
If the prosecution has been able to prove everything they alleged and the judge decided to make an example of Swartz it might have been up to 7 years, but that is unlikely. Swartz's attorney said that if they had gone to trial and lost he thought it was unlikely that Swartz would get any jail time.
[1] https://www.popehat.com/2013/02/05/crime-whale-sushi-sentenc...
6 months in prison is extremely inappropriate for what he was accused of. Threatening far worse to compel someone to accept a sentence they don't deserve is a crime orders of magnitude worse than anything Swartz was accused of.
Crimes should be punished, no matter how little you believe in justice and/or the rule of law.
The life-preserving choice he had every chance to make was simple, no matter how little you believe in the righteousness of admission of clear guilt and/or simply saying what you must in order to move on with life.
These things are all true regardless of your beliefs – or my so-called reprehensibility and/or moral degeneracy.
https://www.theinformation.com/articles/openais-board-set-ba...
You can absolutely get hauled in front of a judge for fraud and abuse in the use of NIH grants. You can be required to pay all of the money back, among other punishments. Google will reveal a variety of cases.
It is in fact rare for retired federal bureaucrats to become professors. It is much more common for people at the Secretary level to get teaching positions at schools of public policy. I am not aware of any former CIA directors holding those roles but I invite you to look.
These schools are not “hotbeds of spies” - truly that it is a completely uninformed statement.
By the way, very few law professors work via grants and even fewer via NIH grants.
I get it. The world is complex, and grouping helps make it more manageable. However, in no way are the "Feds," part of the same tribe, especially in 2023.
>All such schools are hotbeds for govt spies, other folks.
Maybe recruitment? I'm not familiar with the curriculum at USC. Seems like you'd get them when they finished their degrees, unless they offer "wetwork" as an elective.
>They use the same techniques that this neuroscientist uses.
Disagree. Whatever tribe or faction these people you describe belong to, they are definitely not publishing papers. They'd be administrators. They are not involved in prosecuting academic malfeasance, and certainly the prosecution, is not taking advice from retired government employees. He/she/they have their own careers to consider!
Another type of mini-fraud: you could say almost 80% of applied machine learning papers contain serious methodical mistakes that are almost equivalent to falsifying data, and what's worse the authors are well aware of and even the reviewers. I am talking about the kaggle-type ones where you apply some kind of PCA or neural network on some relatively small dataset.
Nobody cared when he was actually killing Iranians or domestics. Back then, kids of exiled Shah-regimers were protesting US gov't support for Iraq (widely reported overseas, widely ignored in the US) at colleges in CA, Michigan and Texas and getting beat up for their trouble.
Plus we were rolling in all that $15/b oil with the flooded oil market so domestic win-win. Except for all those oilmen in TX who went bust. Well, they voted for Ron, so I guess they got what they wanted.
There are fewer of these than you'd think. Modeling wobbling plates is easier mocked than done, for example.
Taxpayers shouldn't be funding studies into the weaving techniques used by pre-Columbian natives in the upper Mississippi basin.
Creating a bilingual dictionary for an indigenous language spoken by a few thousand people who aren't even in the country funding that research is the definition of "a waste of government research grant money."
I feel profound sadness and pity for anyone with such an impoverished worldview. I hope someday you can experience some literature, art, nature, or human connection that can pierce though it and help you reconnect with your innate capacities for empathy and wonder.
Not wanting to use taxpayer money to fund your friend’s whimsical intellectual odyssey is not the same as lacking empathy and wonder.
Keep your pity—you sound like you could use a fair dose of empathy yourself.
It'd actually be great to have some politicians who ridicule frivolous academic research. It's not like there's a shortage of it especially in the humanities and social sciences (where there's no such thing as basic research anyway). Alas, their attentions are all elsewhere.
Did you mean "applied research"?
>It's not like there's a shortage of it especially in the humanities and social sciences (where there's no such thing as basic research anyway).
Careful where you cast stones, my friend. There is a lot of bullshit in software engineering, too.
Gestures towards pile of discarded JS frameworks...
In fact, I'm sure you've noticed the heaps of government-funded software trash.
Academia justifies itself by claiming it's necessary to do "basic", "foundational", "fundamental" or "blue sky" research in a non-commercial setting, because supposedly corporations can't afford to do long term research that doesn't have immediate application. So in this model academic scientists develop a foundation of firm and rigorous theory about how nature works, publish a literature on it for the good of all mankind, and then shorter term applied research is done by companies to derive practical insights from that literature.
That model doesn't seem right, because there are lots of corporate research programmes into fundamental theory that were/are lavishly funded despite a decade+ of no ROI, most obviously at the moment AI. But let's put that aside and pretend it's true.
Problem is that humanities and social studies don't produce a body of theory on which anything can be built. Humanities produces only critical theory, which despite the name is more like a belief system than a scientific theory. The social studies produce a lot of that too, but even outside critical theory their output is just a giant pile of random thought bubbles and mini-studies to try and prove them. There's nothing much linking these studies, no frameworks that make reliable new predictions. Instead we get an endless stream of Just So claims that fall apart when people try to replicate them, but even failure to replicate doesn't have any impact because there's nothing underneath them to be invalidated. At most it's shown that this one specific paper isn't reliable.
Contrast to something like physics where the work of academic theorists is all about coming up with unified theories of everything, which could then be used to develop practical applications. Einstein is super famous because he invalidated large swathes of theory and opened up space for new theory to be developed. Social studies don't have any equivalent to Einstein and can't by design.
The Federal Aviation Administration was named for spending $57,800 on a study of the physical measurements of 432 airline stewardesses, including "distance from knee to knee while sitting", and "the politeal [sic] length of the buttocks."[19]
Office of Education for spending $219,592 in a "curriculum package" to teach college students how to watch television.
Amusingly it does seem to have hit the mark. A former NSF director even lied about the awards to try and discredit it!
Too bad this is all so long ago. I was asking more about the present day, as it seems like that was the implication of the comment being replied to. Where are all these populists with their modern version of the Golden Fleece awards?
From where I'm standing (I hold a Ph.D in a scientific field, and my spouse is a tenure-track academic), an unsustainable portion of academic work consists in justifying one's research program, on grounds that have little to do with the subject of study.
It should hopefully be obvious that I'm not trying to stir a DEI flame war. Rather, I can't shake the feeling that DEI activism isn't accomplishing much beyond creating bureaucraty, and that this benefits neither research in general, nor marginalized groups trying to establish themselves in academia.
Anyway, this seems like a specific instance of a larger trend you're pointing to. Possibly not even the largest or most absurd one.
It's like the same people put application and social consciousness first in their RFPs and then turn around to accept invisible fig leaves during evaluation... Who's getting fooled, the government? Politicians? The public?
Is it, though? In the case of DEI, it appears to me (on the surface, anyway) that these initiatives are being implemented at the request of very vocal special-interest groups, and at the orders of the politicians who represent them. It seems like a fairly straightforward case of lobbying by institutions whose existence depends on the existence of a social problem. It's a rough analog to Eisenhower's Military-Industrial Complex idea.
The same is true with what you call "questionably applied number theory". The only difference is that special-interest group isn't defined along ethno-racial lines, but instead comprises various bean-counters and ad-men who sell various forms of investment.
In my view, there used to be a consensus that such lobbyists were not welcome in universities, or at least not in droves, because of their tendency to subvert intellectual pursuits in favor of short-term profits.
Thanks for the conversation. This has been interesting :)
That's not fraud. Not even the most gullible reviewer believes a word of it.
Leave it to anthropologists of science to describe the ritualistic aspects of writing abstracts.
Some practical ideas, when allocating the money for research 30% or so has to be kept back for replication. Public science is mostly government funded. And the government could allocate X% percent of the budget for replication work. It sounds like a solvable problem to me.
Based on the recent Ranga Dias fraud, I'd think replication culture still has a long way to go.
If the foundation established by the prior research is flawed, attempts to build on it will usually fail.
Because sometimes they can afford to build on your prior results, and advance the state of the art. Factoring in all of the malfeasance, what's the trade off between not publishing due to inability/unwillingness to replicate, and publishing bad results? We should know this tradeoff before significantly changing the status quo.
> or knows how to replicate the thing you did.
Agree completely.
Of course they can't replicate things like ultra high energy particle research, but these sort of obscure things make up a very negligible chunk of all science produced, even if it's quite an important little chunk.
For example in on of the research areas I'm familiar with (optical communications), there are maybe 10 academic labs in Europe (and even less in the US) who have the equipment to reproduce some of our experiments. In our lab there is 1 PhD student who could pull off reproducing the more sophisticated experiments (because he is the one focusing on communications) it took him 2 years to get to that stage.
This is an relatively easy area, i.e. equipment is largely off the shelve, very applied with lots of industry involvement. There are plenty of experiments published which could only be done in 2 labs (both of them industrial), just due to the cost of the required equipment.
In other areas (e.g. with fabrication in the clean room) reproduction would require even more time investment.
Don't get me wrong, reproducing results is important, but what people don't realise it happens all the time when people do adopt part of published results into their research. Mandatory reproducing results would just create large overheads which would get us nowhere.
I suspect you are likely grossly underestimating the available supplies at many research university in the US. For instance things like class 100 clean rooms are basic facilities. Many (and I want to say most) research universities also have partnerships with (if not ownership) of various specialized labs in the surrounding areas for more specific purposes. For instance the NASA Jet Propulsion Lab is managed by Caltech.
Realtime oscilloscope at least 4 channels > 50 GHz bandwidth $0.5M (for some research you need >=12 channels so multiply that number)
Arbitrary waveform generator 4 channels > 45 GHz bandwidth $0.3M (again you might need more than for channels)
RF amplifiers, electro-optic components etc. easily cost $2000-$5000 each and you need several (4-8 at least) of these. The RF cables and connectors/adapters easily cost several thousand $ each.
Fibre components, subsystem components (e.g. a WSS of which you will likely need 4 or so is $50k).
And I will certainly not let a student without training touch the sensitive high-speed RF equipment.
Regarding your comment on clean-room. For many fabrication purposes class 100 is not sufficient (also calling it basic facility is quite rich). And the equipment in is very expensive, LPCVD machines, E-beam, other lithography ... is $10sM. Most universities I'm aware of require fees (typically paid from grants) of $10sk per year to use the facilities (those are the reduced rates for university staff). The training/certification on the equipment typically takes about 1year.
Regarding JPL, yes it's managed by Caltech, what do you think NASA will say if Caltech professors will ask for a student to use the facilities to verify some paper? Sure, lets delay the next Mars mission a year or so, to let some PhD students try stuff in the labs.
I think you seriously underestimate what the cost of using all that equipment is and how much training is involved to be allowed to use it. You definitely don't want any
It's difficult to really explain how much money is spent in top US universities. It's as if there's a fear that revenues might manage to exceed costs. But one of the practical benefits of this is that bleeding edge hardware and supplies, at costs far greater than anything you've listed, is widely and readily available.
Primary research, pioneering new techniques and equipment to explore the unknown, is time-consuming and costly and requires a lot of original thought and repeated failure until success is achieved. However, reproducing that work doesn't involve much of this. It's taking the developed methodology and repeating the original work. That may well involve expensive equipment and materials, and developing the technical expertise to use them, but that does not involve doing everything from scratch and should not take anything like as long or cost as much.
I also believe that we far too readily overestimate the specific special skills which PhD students and postdoctoral researchers possess. Their knowledge and skills could likely be transferred to others in fairly short order. This is done in industry routinely. A PhD student is learning to research from scratch; very little of their expertise will actually be unique, and the small bit that is unique is unlikely to be difficult for others to pick up. I know we don't like to think of researchers as replaceable cogs, but for the most part they are.
My background is life sciences, and some papers comprise years of work, particularly those involving clinical studies. However, the vast majority of research techniques are shared between labs, and most analytical equipment is off the shelf from vendors, even the very expensive stuff. Custom fabrication is common--we had our own workshop for custom mechanical and electronic parts--but most of that could have been handled by any contract fabricator given the drawings. And the really expensive equipment is often a shared departmental or institutional resource. Most of the work undertaken by most of the biological and medical research labs worldwide could be easily replicated by one of the others given the resources.
Depending upon the specific field, there are contract research organisations worldwide which could pick up a lot of this type of work. For life sciences, there are hundreds of CROs which could do this.
As one small bit of perspective. In my lab a PhD student worked on a problem (without success) for over a year. We gave it to a CRO and they had it done in a week. For less than £1000. The world is full of specialists who are extremely competent at doing work for other people, and they are often far more technically competent and efficient than academic researchers.
Sadly, as long as the tyranny of publishing exists, the other researchers will always prefer working on their own things than replicating someone else's not yet published experiment.
Whatever it is, know that science as a whole is giant, diverse, and self correcting over the long term.
When you have lots of people whose livelihoods depend on the gravy train, who can't be sure that what they are working on is fraudulent because they are so specialised, who would take that risk?
Its all about funding. And in the US basically all funding comes from the same source - government, military or corporations - what I think of as the governance system.
Just like we should place failed results (when due to wrong science, not bad skills) on an equal level with successful results, we should place failed thesis projects on equal level with successful thesis projects as having added to the general knowledge (again, when the failed projects demonstrate a falsified theory, not when they fail due to mistakes or inability on the part of the student).
Dan Ariely and Francesca Gino were two of the most well-known behavioral economists. Hell, Ariely even published a a board game about it (as well as a bunch of popular books). They've both been accused of data manipulation this year.
My friends in the field are worried that the whole field will be tainted. It grew out of a controversial idea - that despite what past models demand, people don't consistently behave in a rational way. If the two biggest practitioners of a new discipline are outed as frauds, what does that do the the reputation of the discipline as a whole? Will people be skeptical of any behavioral hypothesis that bucks tradition because Ariely was accused of fraud?
And they are far from the only ones: Diederik Stapel, Brian Wansink, the list goes on and on. (I'm not listing the many names, also at R1 universities, whose verdicts are still "in process.")
What NPR liked to call "replication crisis" was a combination of junk science and blatant fraud.
The whole field (slightly more broadly, of social psychology) is badly damaged for at least a generation.
Pete Judo has some consumer-friendly youtube videos on the topic.
Even when a paper goes through a vigilant, rigorous peer review, it relies upon the data that the research team supplied. Over and over again cases like this have encountered manipulated data. Humans are flawed creatures, and if you spent a lot of time and professional credibility on a hypothesis, there is a strong motive to find what you sought. Doubly so if the results are salacious or contrarian in some way and thus get viral attention. Just convince yourself that it's the data that was wrong somehow and you know your assumptions are right so just this One Time you need to do a little manipulation.
A study I have seen cited on here countless times is the "honesty pledge" one by Ariely et al. It was the one that claimed that when a person signs a form at the beginning, they're more honest. It was complete and utter BS, based entirely on fabricated data. It joins an infamous list of studies that have had enormous influence (especially if they have an "aha!" factor -- if it is the sort of thing that Malcolm Gladwell would talk about, consider it suspect) but were the creation of someone making up data in Excel.
IMO the researcher still has some responsibility because ultimately, it's their research. So the questions to me are:
1) How much due diligence is reasonable? Does it change depending on the source? For example, is it more/less reasonable to accept government-provided data at face value vs. data collected by an undergraduate?
2) What processes can be implemented to safeguard data manipulation? I know there is a movement to provide data with peer-reviewed submittals, but it's still a low probability that a peer-reviewer has the time or inclination to really dive into the data to assess the claims.
The more interesting thing though is why he chose to investigate this question in the first place and why he chose to do fraud to make it seem true. The hypothesis is a very weird one and there's no reason to think it would hold. Unless that is you think of people as being child-like lumps of Playdough, so easily manipulated that trivialities like where exactly something appears on a form can yield huge behavioural differences.
That belief is the only reason you'd ever come up with such a hypothesis, and I think it's not really surprising that someone like that would engage in fraud. After all they have spent months (or years?) on trying to prove that people's levels of honesty are trivially controlled by psychologists like yourself. If you believe that's true then why wouldn't you commit fraud? After all you can easily manipulate people into not noticing it.
Just conjecture of course, but this came at a time when governmental “nudges” were very en vogue. I could see where successful research could be thought of AAA a pathway to influence, prestige, and money through government grants and appointments. And there were some highly regarded behavioral psychologists who were substantiating its effectiveness.
Yeah, governments love the idea that they can influence the population via simple tricks. That's understandable.
Unfortunately nudges are still very much en vogue. COVID was nothing but endless nudging, maybe more like pushing, with tricks like making everything into a social responsibility towards others being deployed endlessly even when not supported by the underlying facts. It worked extremely well. That said, I'm not sure you need psychologists to tell you that "do it for your grandmother" is a powerful manipulation tactic. A lot of the valid findings in psychology are obvious, and the non-obvious findings are often invalid. So we could just defund that field and not lose much IMHO. I say that as someone who has studied psychology. I have a good friend with a PhD in it who thinks the same.
What I found was "How to throw away data that doesn't support your desired conclusions," for the most part. "Actuarial Science," a different field, had some useful techniques but not many. They're most interested in ensuring the bad data doesn't get into the tables in the first place; but at least they are doing "data on data" comparisons and not "data to expectations"
We're building "AI" right now but think about the inputs those see: The very first step is to throw away the statistically too common "stop words" ...
What exactly are you referring to here? This seems like a wildly misguided characterization of statistics, which I am sure cannot be based in expertise or practical applied experience.
> We're building "AI" right now but think about the inputs those see: The very first step is to throw away the statistically too common "stop words"
This is a fundamental misunderstanding of what a "stopword" is and how it's used.
Words like "the" are hard to utilize within with a bag-of-words model specifically. Removing them is not something people do/did because they are clueless monkeys. The goal is to improve the signal-to-noise ratio.
For example, traditionally spam filtering uses a very crude variety of bag-of-words model called "Naive Bayes", in which we assume (wrongly of course) that word choice is completely random, and that the only difference between spam and not spam is that random distribution of words. Are you really going to argue that the word "the" is critical to that process? If you can build a better NB spam filter by including stop words, by all means go ahead and do it. But both linguistics and decades of success in the field are against you.
On the other hand, words with grammatical function like "the" are absolutely important and relevant to the overall structure and meaning of a document. Therefore, training pipelines for modern deep-learning-based LLMs like GPT don't remove stop words (as far as I know at least), because the whole idea of a stopword doesn't make sense in a model like that.
I want to be respectful here, but it sounds like you took a cursory look through three vast literatures, without the perspective of having actually used any of this stuff in real life, and drew some invalid conclusions.
Thanks!
Many people in these fields agree my conclusions are invalid. I say the same about theirs.
I'm the guy who builds the experiments on a team of user researchers. There are all sorts of things that seem intuitive to an outsider but are poo-pooed by practitioners as unethical. For instance, you might run a study that doesn't have enough participants to have a statistically significant conclusion. An outsider would deploy it to more participants to see if the trend becomes significant with more data. A trained researcher will cringe at that proposal.
So far as I can tell, researchers consider the experiment final as soon as you peek at the data. If you want any changes - more data, different demographics, etc - you have to throw out everything and start over. Even though it's logically interchangeable, the data you've already collected is considered spoiled, because they don't want allegations of tampering/data grooming.
A coworker had a saying:
"Data is like a prisoner of war. If you beat it around enough, you can make it tell you almost anything."
I could have made a mistake, or I could have been malicious. I don't think they would have caught it because it would have involved hours and hours of work on their part.
I'm quite certain a huge share of "results" are due to bugs. Probably many of my own too even though I stress about this constantly.
An intentional bug would be practically impossible to show to be intentional. With notebook/REPL style analysis there wouldn't necessarily even be any documentation of the bug. I'd wager it actually happens, and even surprisingly often. We only know of fabricators who are bad at fabrication.
But his papers got thousand of citations as I understand ... that means thousands of people read them?
ahah, if only that were the case :)
"read" is a pretty loose term. I think its a house of cards. When you cite something, you do so (usually) because it supports your paper, basically "X did Y and we need Y to be true for the foundations of this paper." When you cite "X", you do so with the assumption that X did their due diligence and peer review would have caught any issues... but its still only an assumption. If you had to re-create every experiment for every paper you cite, I'm not sure if one would ever actually finish their own research.
I have only ever published one paper though, so take what I say with a grain of salt. It's just my experience.
It doesn't, but that's counter-intuitive.
With relatively new, relatively small citation counts, the numbers are probably indicative of the number of actual readers.
But well established, high citation number papers often take on a "shorthand" role. You'll often see them in introductory sections or other supporting text with statements like "previous authors have X", "common approaches such as Y", etc. Here they often have little to do with the core of the paper, they are providing context.
Now really people should have read them, but sometimes Jones, et. al. 1998 just becomes a collective shorthand for a set of ideas. As such people will quote it just because the papers they did read quoted it, etc.
Often, over time, a single paper becomes the landmark for a set of ideas, and just gets cited to pull those in by reference. In theory this is the paper that "invented" those ideas, but in reality it's more complicated. Overall it's not a terrible practice, as a way to frame things, but can be error prone.
This is kind of why it annoys me a bit when I hear people harping on about trusting science, most science is not as simple as finding objective truths and just reporting them. That's not to say all science is bs and you're better off consulting a magic 8 ball, just that it should never be discouraged to look at methods and conclusions with a critical attitude. There is room for things to be fudged or pushed and very strong incentives for people to do it given how much money and prestige are on the line. It doesn't even have to be as big as a drug trial, one high profile publication can be enough to make a career so you can see how tempting it can be to just change a couple pixels in an image to boost a theory you earnestly believe is true
At the modern complexity and importance of science, we do need to put much more emphasis on checking whether it is done right. The original vision of the Royal Society involved rigorous peer review, to the point of demanding scientists to demonstrate their experiments in front of their peers: https://www.sciencemuseum.org.uk/objects-and-stories/17th-ce... The curator of these literal peer reviews held the first paid position in science.
Journals do have the profits to hire or otherwise pay peer reviewers full time. They would become experts in their field in the process, and thus more suited for the task than the ones in the current system, where even undergrads are asked to review for high-profile outlets like NeurIPS. Also, full-time rigorous peer reviewing would be an interesting career prospect for many current scientists. And here is your startup idea..
The publication volume is just way way way too high. But researchers who don't publish multiple articles per year, regardless if you have anything of value to publish, perish. If you don't churn out paper-per-year in your PhD, you don't get the PhD.
Much of the manuscripts that get submitted to journals are incredibly bad. Most are just bad. But as both editor and reviewer, I usually let them be published out of pity if the stuff isn't blatantly wrong; the poor PhD student's whole career is on the line.
Journals being full of crap is not SO bad within science because everybody knows they are full of crap. But if an "outsider" thinks that being published in a peer-reviewed journal, even a "good" journal, means that the article isn't crap, it can be literally life-or-death (like in this case).
Academia is broken because it's being run as a business whose purpose is to churn out papers. Welcome to neoliberalism.
1 per year is a ridiculous standard.
In Finland usually at least 3 peer reviewed (first author) papers are required for a PhD in my field (cognitive science). In some fields (e.g. many engineering fields) even more. And PhD grants are typically for three to four years.
It is ridiculous and the paper quality is what you'd expect.
It's also true that publication rates post PhD vary wildly as well, e.g. expectations for a tenure packet.
However, it's also fair to say that expected publication rate has significantly grown universally. A generation or two ago, a solid career could be built on a handful of high impact papers. That's hard to imagine now.
You can write a monograph (essentially a book) instead but that's frowned upon. Especially by admin because the papers bring the univerisity more money than monographs.
Nowadays piles of inconsequential papers where you contributed little but your name is almost a requirement for a solid career. It's horrible.
During those days, the science you were talking about was one of countless hobby clubs for aristocratic circles, where bored rich folk could dedicate themselves to one upping each other in the discovery game their peers and recent predecessors had invented. There were certainly cheaters then, too, but the whole thing was very insular and the only people who even cared about it were the people who were in on the game themselves. So excelling within the rules and elaborate demonstrations of the game were all part of the club sport.
400 years later, billions of people across five or six generations have been told that this game is the secret to human prosperity on earth and that the more its played, the more aligned we'll be with the truth of the universe, the longer we'll all live, the more leisure and luxury we'll all enjoy, etc
That's a whole different game! The demand for scientific output has gone from an exhibition sport shared among a small and snobby circle to a replacement for an eroded theology. Trillion dollar governments and globe-spanning trillion dollar industries now (ostensibly) make their billion dollar decisions based on each day's summary of the sport, and billions of people dreaming about prosperity or salvation await the next big game's result as though it were a demonstration of divine grace.
There are now so many games being played, by so many people, with so many bets and spectators, that incentives are unfathomable and referees are sparse and cheating is rampant.
Unfortunately, you're not going to clean all that up by telling some journals to reshuffle their revenue allocations.
I think far and away most scientists become researchers despite the fact that it’s generally not at all a prosperous venture.
Science is seen as the road to human (not just personal) propsperity by people educated in the last 100-ish years, which is mostly everybody now, which puts extreme pressure to perform/produce on what was once a pure little hobby sport whose spectators were almost all invested as players themselves.
After PhD there will be nobody telling, and often not even caring, what to do. But that may mean that you don't get your PhD or you don't get another grant to live on. If you get a tenure it almost literally means that you can't be fired even if you do nothing at all. What is surprising is that almost all with tenure keep running the rat race even though they don't really get anything at least material out of it.
(Nitpick: I think serious research is the fun one. The one churned to get another grant is neither serious nor fun.)
> After PhD there will be nobody telling, and often not even caring, what to do.
If you make the hiring cut in you’re in for about 5 years of grunt work and committees as an assistant prof, right?
By “serious research” I mean the one they do for career advancement and hiring.
I am a postdoc, just have been for quite a while (on four different grants at least). In Finnish academia it's not that uncommon to stay a postdoc even until retirement.
> That’s not true. If you make the hiring cut in you’re in for about 5 years of grunt work as an assistant prof.
For teaching and admin yes. But at least in fields I know, what research you do or whether you do at all is all up to you. Of course the risk is that you'll be unemployed after the assistant prof. term ends. My point is that if you don't care about that, you're quite free to do whatever research-wise.
If you factor in all future humans in your utility calculations, then science is by far the #1 noblest pursuit humans can ever undertake. We're talking about scientific advancements potentially helping trillions of people before all is said and heat-deathed. Great works of art can also be enjoyed by all future humans, but science has a super-linear (if not actually exponential) growth curve where every advancement makes future advancements a little easier.
The question is whether the noble pursuit of "science" and the day-to-day activities of "being a scientist" have diverged or not. There is mounting evidence that the institution of science has been subverted to the extent that many people who are professional scientists are not actually contributing to the pursuit of science. Or, far worse, detracting from it, as we see here. It's one of the great tragedies of our era.
How exactly do you know this civilization will not crash too? (and take with it that highly localized habitat)
Everything is temporary on a large enough scale and everyone w/i that golden era also thought it would last and that belief is part of why it didn't.
Humans will never build anything that truly lasts for a very simple reason. The lessons learned get forgotten after 2 generations (3 at most). Stop and consider how the US fought for its freedom and how nowadays many people from the US would vote to have more limitations on speech.
What are you talking about? Did we all forget the Pythagorean Theorem after 3 generations? Did we forget the force-multiplying effect of levers and pullies a few generations after Archimedes died? How can you sit here with Wikipedia at your fingertips and tell me that the collective sum of humanity remembers nothing from over 100 years ago?
> everyone w/i that golden era also thought it would last and that belief is part of why it didn't
Yet from almost every collapsed golden era, scientific progress from that era made its way back to collective knowledge (sometimes very slowly, admittedly). People in this modern age have this extremely simplified view of what "collapse" actually looked like.
What's the difference between data and information?
data is data
information is data with context
The values and lessons learned to build something that truly lasts gets forgotten over time until the thing that was built gets changed into a weaker form as the people involved stop valuing the things that gave it the stronger form.
The vast majority of established science will outlive mere nations and petty politics. There are too many copies of Wikipedia, too many printed textbooks and encyclopedias, to lose a significant portion of established science. The possibility of worldwide humanity-destroying events does not disprove this at all. In fact, if we ever faced a humanity-destroying event, then it would be nothing less than our collective scientific progress that would have any chance of seeing us through it.
What do you mean by many? As an institution, I sincerely doubt that science is exploited to a similar degree as basically any others.
I don’t really accept these ridiculous trumpisms. “Many people are saying”.
How many people are *doing*? I’ll wager it’s far and away a major minority.
Can you back up your “many” with actual numbers as a percentage of investment?
The world being shitty in other ways does not diminish the tragedy that we are only a fraction of how effective we could be at pursuing science.
> Can you back up your “many” with actual numbers as a percentage of investment?
"A 2011 analysis by researchers with pharmaceutical company Bayer found that, at most, a quarter of Bayer's in-house findings replicated the original results."
"In a 2012 paper, C. Glenn Begley, a biotech consultant working at Amgen, and Lee Ellis, a medical researcher at the University of Texas, found that only 11% of 53 pre-clinical cancer studies had replications that could confirm conclusions from the original studies"
"The ... paper examined the reproducibility rates and effect sizes by journal and discipline. Study replication rates were 23% for the Journal of Personality and Social Psychology, 48% for Journal of Experimental Psychology: Learning, Memory, and Cognition, and 38% for Psychological Science"
https://en.wikipedia.org/wiki/Replication_crisis
> I don’t really accept these ridiculous trumpisms. “Many people are saying”.
I didn't say "many people are saying", and comparing me to Trump is a much greater insult than things comments get flagged and removed for. Respond to what I actually said and avoid the (extreme) personal insults.
I did my PhD under a professor who was very respected and influential in his field. He published quite rarely, although he had piles of manuscripts that would have passed review but he didn't find them worthy. And refused to have his name on his lab's papers if he didn't feel like he contributed enough (nowadays many/most professors who won't even read the paper require them to be added as an author).
In the current system he would have no chance in academia. He said so himself. As has Peter Higgs: https://www.theguardian.com/science/2013/dec/06/peter-higgs-...
As I was reading the article I got to the part where the whistleblowers accused zlokovic of pressuring them to change data.
If you changed data then you're also responsible. Every one of those whistleblowers should have refused.
I agree they should have refused. I hope I would in their situation.
I see this meme about the early scientists a lot. Unfortunately, these first early scientists are not so easily categorized. I would encourage others to delve into the biographies of these progenitors. It is true, some were very much in the vein of this meme. But many were much more complicated individuals.
For example, Darwin is about as blue blooded as it comes. Yet Origin of Species has a very long section at the beginning where Darwin painstakingly goes over all the scientists before him that that in any small way had already discovered evolution.
Another good one is that while digging potatoes with his hands at the family farm in his native New Zealand, Rutherford got the news that he had been awarded a scholarship to study physics at Cambridge under William Thomson (Lord Kelvin).
Many other scientists came from very 'low' births. But science is a 'strong-chain' domain where only the 'correct' ideas survive. Anyone can, and did, contribute despite those obstacles.
In medicine it's especially tough, as testing is among the most expensive and the number of available data points is quite low, so one gets to fignt over individual data points that make a difference. Did this person not follow the protocol, and I have a paper, or not?
I've seen something similar when working in agriculture: Early tests of new plant strains are low data, because a company will start by testing so many experimental plants that it'd be unaffordable to test them all very strongly. This makes people really argue about single data points in those early tests. Was this chunk of the field contaminated? Attacked by a wild animal? But there at least the interest in fraud is small. If a breeder gets a stinker through, it just goes into trials on the other hemisphere, where it's planted in an order of magnitude more fields, and therefore just lead to disappointment 6 months later.
With medicine, the cost of replication is so high, and often takes so many years, that it's not just that an honest mistake is catastrophic, but that the difference in outcomes for the person doing the curation is so high, one doesn't have to be all that dishonest to make biased decisions that will lead to a strong career.
The wider science needs to learn a lesson from the Economics science: Incentives Matter. A lot.
Incentives matter a lot but creating incentives that work is almost impossible.
Science used to run on ethics (like many other professions). But we learned from economists that there are no such thing, only utility (money) maximizers.
Fact is, corporate R&D doesn't have this relentless problem with reproducibility. It's academic output that does, because academics only care about getting a paper published and don't expect that anyone will use their results. Often they don't even make their code or data available at all because it's not to their advantage for others to be able to replicate their work, as that would yield fewer papers. But this is all wrong. Science exists to be used in technology, not for its own sake.
I've worked for two corporate labs and have collaborated with several. There was less rigor if anything.
Corporate lab work is quite cozy and stable. In corporate lab your job doesn't end if you don't (pretend) to get a new major discovery every year. They don't make anything available, often not even internally.
Science is about a lot more than technology.
Maybe the problem is that there are many corporations, with different cultures.
But there is mostly just one academic culture across the whole system.
Whereas with academics, you get a paper. You might get data and code, or might not, depending on field and temperament of the researchers. If you're really lucky that data/code might actually be correct, match the paper and be usable for something, but it really depends a lot on the field. You almost certainly won't get products.
I don’t know about this. (But I don’t have any answers for the question either).
If you’re a full time peer reviewer, first, are you really a peer? But more importantly, your motivations change. No longer are you looking to try to see if the paper is worthy of publishing or if it is sound; instead, your motivation is to push through as many papers as possible. When getting paid depends on approving papers, the quality will drop.
Maybe the problem is where the money exchange occurs. What about if authors paid to have their paper reviewed, instead of published? Currently, journals only get paid when a paper is published. What about if they got paid to review the paper at all? It would limit the paper submissions to Nature/Science/Cell, but you’d be paying for a high quality review (which often makes a paper better). You might even have luck with decoupling reviewers from journals completely… make the journals compete over the best (already) reviewed papers.
I think this is an assumption that doesn't have to hold true in practice. Maybe it's biased by the 'publish or perish' paradigm that's pervaded academia, but there's no reason to replicate the same problem elsewhere.
But speaking to Science anonymously, four former members of Zlokovic’s lab say the anomalies the whistleblowers found are no accident. They describe a culture of intimidation, in which he regularly pushed them and others in the lab to adjust data. Two of them said he sometimes had people change lab notebooks after experiments were completed to ensure they only contained the desired results. “There were clear examples of him instructing people to manipulate data to fit the hypothesis,” one of the lab members says.
The incentives are clear to me, but punishment is less of a deterrent than good incentives, but which incentives would reduce this allegedly fraudulent behavior?
Fair, but what is the alternative that would actually work? What is the budget of all of the journals compared to the NSF+NIH? Is medical research that is true, and certainly actionable worth as much as an F whatever fighter jet? People will have to decide.
There are some alternatives that I'm aware of. Here's a few:
1) One is to allow journals to focus on less-than-great results. Right now the focus is on novelty, so there is an incentive to show that your work has some new, great outcome. But there's also value in showing "Hey, we thought this idea had legs but it turns out it didn't." Publishing that work should be part of science but right now its not. (As a side benefit, you could prevent a lot of researchers wasting effort on the same idea simply because they weren't aware that other people already tried, and failed.)
2) Journals can put a premium on sharing your data and code during the review process. Right now, it's often just up to the author and there are lots of veils to hide behind that essentially give the impression of sharing data, but not in a very useful way.
3) Give value to replicating work. Maybe not as much prestige as creating new work, but showing that it can be replicated obviously has value to society as a whole. Most of the time this won't get published, except in the cases where it's sensationalized, like fraud. (This effect is related to #1)
4) Journals can do a better job vetting their reviewers. They struggle to get timely reviews and reach to anyone who accepts the duty. Reviewers may agree to review something they have little background in, and as a result, it's easier to skirt bad articles through the system.
At some point the prestige & funding are just too much of an incentive to not take advantage of it.
It's probably easy to morally justified in their minds because they believe their hypthosis is correct
But most agree there is benefit to having sport activities in our society. And that doping is detrimental to it.
It also attracts millions of taxpayers money, lots of athletes, especially those going to Olympics, get funded.
But originally I wasn't making a comparison based on negative impact on society, but comparing the incentive structure.
When people say "trust the science" it means making scientists and researcher High Priests of the Religion of Truth. Until every single experiment is pre-registered, all data is public and transparent, and all results are published, the entire experimental scientific establishment should be treated with massive skepticism.
The topic is related to the quality of science. Science is composed of scientists.
>Do these people define what science or is it an established set of principles and methods?
Science is defined by the actions and output of the entire system.
Perceptions of science, your beliefs (perceived as knowledge) are affected by how scientists (and their fan base) define "it".
> You think scientists are the only ones who can do science
No.
> and what they do, wrong or not, defines what science is??
Yes. If you think otherwise, I would really like you to walk me through your reasoning.
Maybe you are conflating it with how it is supposed to, and is claimed to function by people who share the same form of faith based epistemology as you.
You still have the title "scientist", and still get your paycheque. Like baking, there is the recipe one is supposed to follow, but there is also the how the baking is actually done. If a baker failed to follow the recipe in an instance of baking, would you also believe that they are not a baker, or are not baking?
I think it's interesting how people intuitively frame (construct a virtual model of reality, and perceive/present it as reality itself) the practice of science such that it "is"[1] literally impossible for scientists to do wrong, and with such a simplistic method: if it isn't perfect, it isn't science (which opens up a serious ontological problem: because it cannot be known to what degree each potential scientist executes the method with perfection, it is not possible to know how many scientists exist, or if a given candidate actually is a scientist...an individual could be one for decades, and then one off day and Shazam: you "are" no longer a scientist, despite having the title, the income, and the respect and admiration, despite not actually being the thing itself).
>Maybe ChatGPT can explain that to you instead of listening to Alex Jones.
What's the current scientific consensus on mind reading? Maybe it's not me who has to brush up on my scientific scriptures.
And since we're on the topic of who to take advice from: perhaps you should reevaluate the trustworthiness of that Oracle inside your mind, because it's "fact" here is way off: I do not listen to Alex Jones. Do you now wonder how many other facts your Oracle got wrong? My Oracle suspects not, but cannot be sure.
[1] here I am using the colloquial, normative meaning of the word "is": how humans believe "reality" "is".
You seem to be completely obsessed by titles for no apparent reason. I don't care what your title is, if you fake data to validate false hypotheses you aren't doing science. It's very simple.
>You still have the title "scientist", and still get your paycheque. Like baking, there is the recipe one is supposed to follow, but there is also the how the baking is actually done. If a baker failed to follow the recipe in an instance of baking, would you also believe that they are not a baker, or are not baking?
If you purchase a cake from Walmart and tell people you baked it from scratch you are not a baker. If you 3d print a cake look alike made of plastic and tell people it is a cake you are not a baker.
You seem to be in the midst of a mental break so good luck to you.
> You seem to be completely obsessed by titles for no apparent reason.
You seem to be an overconfident Naive Realist.
> I don't care what your title is, if you fake data to validate false hypotheses you aren't doing science. It's very simple.
I doubt it. You don't take the opinions of scientists more seriously than non-scientists? Shall I go through your comment history to find instances?
And this is the problem: "science" (which is copposed at least in part by scientists) CANNOT make an error according to this reasoning.
>>You still have the title "scientist", and still get your paycheque. Like baking, there is the recipe one is supposed to follow, but there is also the how the baking is actually done. If a baker failed to follow the recipe in an instance of baking, would you also believe that they are not a baker, or are not baking?
> If you purchase a cake from Walmart and tell people you baked it from scratch you are not a baker. If you 3d print a cake look alike made of plastic and tell people it is a cake you are not a baker.
As the saying goes: Reality is perception (as demonstrated by your very comment!).
> You seem to be in the midst of a mental break so good luck to you.
Do you have any interest in whether the reality your mind generates and projects into the "you" service's experience (as "reality") is actually correct?
For example, take your prior comment:
>> Maybe ChatGPT can explain that to you instead of listening to Alex Jones.
By what means could you acquire knowledge of my interests? Feel free to peruse my comment history, you'll find no praise or likely even mention of Alex Jones (I think he's a dummy, though I do like him). And if you're going to suggest you have mind reading capabilities, I am happy to have that argument.
Could it be, perhaps, that an idea popped into your mind, and you accidentally forgot to apply any(!) epistemological rigour to it before streaming it out onto the page, like an LLM? I mean, come on man.
Some of this has good intentions, at least originally. Weather stations aren't normally intended to be used by climatologists. They exist for other reasons. So they get moved around, or not moved even as the environment changes around them, get placed in inconveniently unrepresentative places like airport runways, and more. Climatologists scrape this data from the internet or collect it from logbooks and then try to work out what's happening, but the data is super noisy.
Now the way science works is that you characterize the uncertainty in your data and propagate it through any calculations you do, in order to track your uncertainty intervals. Then you communicate those and take them into account when making predictions.
But in climatology they don't do this. Instead they use lots of algorithms and manual tweaks to try and "fix" the data to bring it into line with what they know it "should" be, and then report the data without CIs, as having 100% confidence. For example if a time series at a weather station is stable for 20 years, then experiences a short break, then it returns but the average is consistently 0.3 degrees different than before, they infer that it must have moved and they then "correct" it back to the previous baseline. If there are gaps in the data then they generate fake readings by interpolating between the nearest alternative weather stations, and so on.
Outsiders might expect that they would investigate and try to improve the quality of their source data but they don't. Like, if their algorithms infer a station move, they don't contact the station operator to figure out if that really happened. They just assume their corrections are fine and move on.
Another fun thing they do is alter data that was already published. When they update their algorithms for deciding what data points to include/drop/change, they don't just use it for new data running forwards. They reprocess the entire historical data set. That can yield outcomes that would normally be taken as a clear indicator of scientific fraud, for example where NOAA declared a temperature record, and a few years later declared a new record that was lower than the previous one [1]. Or where scientists invalidated decades of published papers (thousands of them) by deciding that the temperature trend in the first 15 years of the century was totally different to what had previously been reported:
https://www.nature.com/articles/nature.2015.17700
The underlying data on which those papers were built was announced to be all wrong, but nothing was retracted! That's how science functions. And the best part is that they've trained the public so well that for anyone who calls any of this fraud, as you just did, they are instantly ostracised for being a heretical Denier.
[1] https://retractionwatch.com/2021/08/16/will-the-real-hottest...
Speaking loosely to make the point, they used to measure temperature from the front of the ship, then inexplicably (to him, anyway) changed the procedure to sample from behind the ship (where the water was warmer from engine exhaust/water output), leading to a sudden increase in water temperatures.
"Zlokovic is recognized internationally as a leader in the fields of AD and stroke research. Thomson Reuters and Clarivate Analytics listed Zlokovic as one of “The World’s Most Influential Scientific Minds” for 21 consecutive years (2002-2022) for ranking in 1 % of the most-cited authors in the field of neurosciences and behavioral sciences. He received [lots of awards.]"
[1] - https://keck.usc.edu/faculty-search/berislav-v-zlokovic/
I.e., this supports the suspicion that policy makers who claim "the science" supports certain policies are drawing on bad information / compromised experts.
Note: I'm personally very pro-vaccine. I'm suggesting that we treat academic misconduct according to the harm it could cause.
The vaccinations were still a net benefit for COVID, but the unexpected side effects have seriously damaged people's trust in vaccines in general. We're going to pay the price for that for decades to come.
See the problem?
People who believe they know what is best for any other person are exclusively idiots.(not saying this is anyone in particular)
Also, please don't suggest the shots did anything. I get sick once per five years. It's plausible that the shots helped but not definitive.
With the above premise, where do you draw the line? For example, are you complicit if you use that electricity (or buy that cell phone made from unsavory practices)? From a legal standpoint, should you be charged because you materially benefitted from a criminal action?
How do you copy the number of deaths that happened because Research_X went unfunded due to fraud occurring in Research_Y?
Besides, you don't even know that the unfunded research would have resulted in increased life expectancy because the research was never conducted. You're trying to prove a counter-factual where you don't have data. Additionally, there are probably far too many confounding factors in health science to make a strong statistical claim against a single action as you're insinuating.
That being said, I think it would be unfair to tarnish all of science with this brush though. There are many fields that don't suffer these problems nearly as much.
When it comes to trials where the results matter (ie. for approving a medecine), the blinding process should blind the scientists doing the trial.
There should be a third party who will mix up the placebo and real vials of treatments. The treatments should stay in a sealed bags until used, and when used there should be two people in the room recording anything that happens.
For treatments involving "inject this drug", the scientists, investors, and anyone else with a horse in the race shouldn't even be present.
The third party is told "please mix up these tubes randomly and don't tell me how you mixed them".
Both the third party and the scientists would need to be dishonest for the scheme to break down.
- Samuel Shem's "The Fat Man" character(IIRC)
Goodhart's Law says: "When the measure becomes the target, it ceases to be a good measure." When the target is "publication or citation count" rather than "meaningful research or the pursuit of truth" things can get weird.
- Publish-or-perish: You want to keep that professorship, you had better publish, frequently. Experiments that don't yield results don't count.
- Grant money: You want to get funding for your lab, your assistants and your students, you had better publish. Experiments that don't yield results don't count.
- Start-up culture: Just another permutation of "get rich quick", but you need a saleable product. Experiments that don't yield results don't count.
So there are massive incentives to fake results. "Fake it until you make it", only with actual lives at risk. Say hello to Elizabeth Holmes.
Money doesn't seem to be at the root of the first one.
Meanwhile people doing actual work get scooped or ignored by people abusing the practice or just doing outright fraud.
There needs to be a journal dedicated to documenting unreproducible science studies and papers.
The inability for scientists to police themselves is the death of Science.
I'm all for the overhaul of course. But it's not fixable solely by publication criteria. In fact publishers benefit from poor quality. The publisher system makes no sense and is actively detrimtal to science and disseminating information.
Everyone wants to say "fraud bad" but no one seems to want to distrust any science that's not validated by a second, unconnected source.
My hypothesis is that fraud will continue.
This looks like a huge case of deliberate malfeasance. Real shame.
Science has a PR problem and like it or not, science needs the support of society at large to succeed and move forward.
What "inside" are you in?
I don't see this as a non-issue as a scientist. It's a massive problem. It means that when I read papers, I'm constantly extremely suspicious of the results.`
This applies to multiple levels in the publication game. Lab grants would likely dry up if the size of the problem was revealed to be massive. Publications would lose credibility, prestige, and money if the problem proved large. It can be recognized as a problem by those who's salaries don't depend on it, but still remain unchanged because those in the best position to change it have salaries that do.
If we are talking about the number of scientists who see flaws and hold skepticism, that's absolutely fine (I'd argue the number could be much higher and be quite healthy overall). But we are talking about integrity. Ethical research, lack of fraud, lack of bias, lack of perverse incentives, a system that produces high quality and trustworthy output by design, etc.
As a scientist this how I always read papers. It's how I was trained. Don't just read the results, read the paper and critique it - was the method appropriate? were control tests performed? is the statistical analysis sound? And even beyond that - if a substantial claim is made, it's assumed to be an anomaly until reproduced by others.
I will say that labs get a reputation. Well known labs with PIs who produce good science are usually treated with a more accepting view (i.e. this is probably true). Unknown labs are treated with a ton of suspicion.
Take masking for example, and here is a bomb most people are totally unaware of:
You want the population to wear masks so that the sick people are wearing masks. Masks aren't going to do much to prevent healthy people from getting sick. But they will do a lot to stop sick people from getting others sick.
This truth is almost never communicated though. Why? Because then wearing a mask becomes a "mark of illness" and nobody wants to label themselves as sick or potentially sick. So the solution to this is everyone wears a mask all the time. But you cannot communicate this whole scenario either because most people aren't smart enough to grasp it or not smart enough to understand why they need to wear a mask healthy or sick.
So this is the kind of problem science faces, and frankly I think the best thing to do is exactly what they did: Insinuate that wearing a mask protects you and insist everyone wear a mask.
I can't help but think this attitude is awful. Most people are too dumb to understand that it is the sick people who should wear masks!? Like what in the world!?
No. You shoot straight with the public. Then you plead with them to do what is best for everyone by wearing a mask when they are sick.
The alternative, as you suggested, is extremely damaging to society longterm.
This doesn’t negate your point, but I know someone who’s an "insider" in this particular scandal and sees it as a systemic problem. This researcher/scientist was very disturbed by this scandal after learning about it, and some of their colleagues feel similarly sickened. If you're a good person, it's pretty disturbing to learn that some of your peers would be willing to risk lives by running a trial based on intentionally faked data.
When I found out a software bug had invalidated a year's worth of simulations they pressured me to present the false data as true at a large talk. That's when I realized that being honest and truthful with data was not the way to succeed in academia. I also unknowingly ended my career that day by not going along with my professor's fraud.
That being said, nothing surprises me anymore about academic dishonesty and research grant funding.
They are not supported on number of hours of research done-- they arguably put in that time when they were PhD candidates or post-docs themselves.
When I was a PhD candidate (couldn't handle the papers-or-die academic environment so I left) I really, really loved my PI. He had immense academic and intellectual humility. But even he had 10+ papers in-flight at the same time and rarely had time to do any work himself!
I think that's actually a good thing. Makes publication metrics unusable (which is a good thing) and sidesteps many gaming strategies.
- There are still internal databases (which funding agencies can access), people fight over who gets on which paper there. There are people trying to game that system as much as they would try to game a public metric.
- There's a whole cottage industry of people in these collaborations who simulate the experiment, show that it could do something, and then publish a paper (with a shorter author list) about it. This is so popular that many of these people never get around to doing the real experiment.
Unfortunately I don't think we've broken free of Goodhart's Law so easily.
- oss-project implements paper in another framework. makes a small mistake to the paper.
- original lab takes the OSS code because they switch frameworks
- they find and fix the mistake
- sit on that for 2 years without doing a PR.
The solution from the PI was to have everyone buddy up and tell your partner where you kept code on your laptop.
Anyone trying to replicate the results will discover that the previous simulations are wrong, though will probably assume mistakes and not fraud by the original researchers. Of course it could take years for someone else to notice the problem (if ever), depending on how exciting were the claimed results and how easy to run are the simulations.
> Two of the insiders also say Zlokovic sometimes had his team improperly alter existing notebooks. Normally these notebooks—in which scientists record details of their work as it proceeds—provide a ground truth for an experiment’s methods and results. As a result, they’re also often central to misconduct investigations.
> But two of the former lab members say that after an experiment was completed and its results published, Zlokovic sometimes admonished his scientists to make sure the notebooks were “clean.” That was understood to mean pasting into them printouts of the published results and methodology or omitting contrary details that challenged the paper’s conclusions. Zlokovic explained that those changes were needed in case of an “audit,” according to the two scientists.
Those allegations are pretty damning if they are true. Falsifying lab notebooks is very obvious scientific misconduct, and there are no benign explanations for that.
At what point do we just call it "lying" and "fraud"?
Trust scientists, even when they lie and cheat, they do it for a higher purpose that the general public can't understand.
I think its time for a reformation.
This is not a situation where a simple correction needs to be done. This guy caused damage and wasted limited time and resources. Sure we can throw out his results but science was still scammed. Losing your career for such egregious behavior is as limited a punishment for this as preiests being shuffled off. This guy should be in jail at the very least.
Universities run a lot of even basic functions on grant money (university takes a cut in money and/or labor) and researchers and labs are the ones that apply for the grants.
For most science there is no obvious commercialization at least in any predictable timeframes. When there is, it's typically called R&D.
It would just make them look worse and make people question why they're giving that VC money.
This is the primary critique from the left of the VC phenomenon.
Turns out human meat suits are just driven by biology to survive and not egalitarian human philosophy and psychology
Commercial sponsors maybe benefitted from the positive results and don't want to risk losing future such benefits by biting the hands that feed them…
NIH and other funders would be the ones that would like their money back (although they don't probably care either).
Everyone who I know who has encountered it just switches who their research is based on. Then doesn't go get smacked down. Quite risky to do so.
but well... people gets what they asked for, so everybody should be happy with the current situation.
If cheaters have the money it means also that somebody has zero money. Is a double loss. This people simply pass over the other researchers and push them out of the road.
---
'A 2012 research paper found that only 11% of 53 pre-clinical cancer studies had replications that could confirm conclusions from the original studies.[79] In late 2021, The Reproducibility Project: Cancer Biology examined 53 top papers about cancer published between 2010 and 2012 and showed that among studies that provided sufficient information to be redone, the effect sizes were 85% smaller on average than the original findings.[80][81] Another report estimated that almost half of randomized controlled trials contained flawed data (based on the analysis of anonymized individual participant data (IPD) from more than 150 trials).[83]
---
There's also a simple pragmatic issue. When finding something is effective gives you billions of dollars in profits, and finding it ineffective gives you millions in losses, you have motivations beyond just the truth. This is where regulatory agencies are supposed to come into play, yet those agencies tend to be staffed (if not lead) by people from the exact same companies they're supposed to be regulating.
[1] https://en.wikipedia.org/wiki/Replication_crisis#In_medicine
Admin people like the principal may get obscene salaries (like 4 times the max professor salary) though.
As an academic, I think academics (including me) are paid too much. We need people who are interested in the science, not those who (pretend to) do it for money.
In the US, the problem is mostly the opposite. Lots of people would rather do science, but it's hard to choose using your skills for real science when some company will pay you 2x-10x to instead optimize ad clicks on their website or algorithmic trading or whatever.
In some of the humanities, liberal arts, social sciences etc academia may pay better than other options for those people. But for most STEM folks, it's a tradeoff between "low-paid meaningful work" vs "high-paid meaningless work".
Medicine and AI research may be the only two areas where people can simultaneously do cutting edge research and make high salaries.
It's really easy to do "shortcuts" in academia that gets one better salary if that's what they want. As seen in this post.
https://transparentcalifornia.com/salaries/university-of-cal...
Offtopic: At least for an european it's totally absurd that the top 10 or so on that list are sports coaches.
https://transparentcalifornia.com/salaries/2022/university-o...