Harvard ethics professor allegedly fabricated multiple studies
thecollegefix.com
thecollegefix.com
Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments)
Committing fraud paid well. HBS professor salary is about $220K per year. Ten years of salary is a $2 million payoff. Big incentives to do this.
Worth a policy change by university for clawbacks of salary and especially pensions and tenure. Like insider trading she should be forced to disgorge all the profits from her fraud. Dan Ariely too.
And worth charging for criminal fraud, she defrauded her university and students. 5 years in jail seems fine.
It's time to stop pretending academics are a medieval nobles who grow faint at the idea of their sacred honor being besmirched. With huge salary opportunities for bold populist claims academics will take big risks to achieve choice jobs even if they think they will eventually be discovered.
As a matter of employment law, it's generally pretty difficult to claw back pay from an employee. And this is almost always a good thing.
I'd assume that such a policy change could actually have severe unintended consequences as it would necessitate some metric to evaluate this, which it is notoriously difficult to do with respect to research. Not to mention that most research that is wrong is not wrong to intentional acts of deception, but just that people make mistakes. I'd assume the consequences would result in this territory, making mistakes far more costly. Not something we want for research, where essentially we're trying to learn how to play the rules to a game by playing the game. Often mistakes help us learn those rules.
No, universities are a reputation economy and do their best to avoid reacting to or calling attention to fraud. They benefit from the perception that their professors are all competent and honest. This particular incident has got some press pickup due to the irony of an ethics professor who wrote a book about breaking the rules at work being caught cheating, but academic fraud is common and people who report it say that almost invariably nothing ever happens. Even in the rare cases when there's an investigation, it all gets swept under the rug. Note that:
• Harvard took over a year to request retractions, even after outsiders did the heavy lifting for them.
• This prof has silently been put on leave.
• Harvard won't comment.
• She has many coauthors on the bad papers. Nothing has happened to them. They were all blissfully ignorant.
> most research that is wrong is not wrong to intentional acts of deception, but just that people make mistakes
In reality the line is blurry to the point of being meaningless. Many glaring problems in research papers get claimed to be "mistakes" even when there's no plausible way they could be so. You have to be willing to penalize mistakes too, otherwise you can never penalize fraud.
As an example, check out the story at the top of this essay I wrote a couple of years ago:
https://blog.plan99.net/fake-science-part-i-7e9764571422
The before-surgery and after-surgery photos are clearly pixelwise identical except for the skin lesions, which were airbrushed out. Despite the sheer visual obviousness of this fraud the journal didn't notice, and when outsiders did the authors simply claimed “The photograph was taken in the same room with a similar environment; unfortunately the patient wore the same shirt” and the journal accepted this explanation! A ten year old could do better than that.
BTW don't conflate the unknown nature of answers to research questions, with the known nature of how to scientifically answer questions. The rules aren't hard, especially when you start from a base as low as "don't fake your data".
As to your other comments and blog, I'm well aware. And if I may add some constructive criticism, if you want to call something pixelwise identical, it helps to show a differencing map. It just makes the claim substantially stronger and clearer, especially since there's a lot of additional factors that can change pixels, making the likelihood of a match even lower. These are things that wouldn't be possible by their explanation, whereas human interpretation is more subjective (it'll be a bit messier than this though). Though even low differences are still pretty damning.
> with the known nature of how to scientifically answer questions.
I think this is a bit naive though. Generally in science we rule things out, not rule in. That's why I'd be hard pressed to say that we have a "known nature of how to scientifically answer questions" because "scientifically" is doing a lot of heavy lifting and is fuzzy. Essentially "there's wrong ways to do science, but not a clear 'right way.'" Obviously faking data isn't okay nor is just doing something and calling it an experiment. Rigor is necessary but what constitutes sufficient rigor is dependent on the strength of the claims and thus the uncertainty of those claims. A big issue we face is that we take empirical results as equivalent to analytically derived theoretical models combined with experimental results. i.e. Physics develops a theoretical mathematical model and tests experiments against that whereas {psych,medicine,ML} make a non-mathematical hypothesis and report results. The problem with the latter is that it is harder to capture uncertainty because it isn't as strictly bounded. But it is still science, and the reason this is done is because the models are likely intractable (if not provably so). But that's why Asimov wrote about the Relativity of Wrong, rather than the precise way in which we can exactly do science. In fact, if the latter was true, we'd program machines to do it. Just a bit more nuanced.
Point well taken. The a-theoretical nature of the social sciences (and many others) has come up before. "A problem in theory" discusses this:
As for science, my main point is that we need to embrace the noise. A lot of people try to use metrics to reduce the noise (which is good) but have this fundamental belief that noise can be absolutely removed. But this is a deep misunderstanding of what noise is. Ignoring the probabilistic nature of the universe, even a fully deterministic system has noise. A good example is the lack of absolute precision. Statistics captures error and helps us analyze that error. But error always exists due to the lack of both omniscience and omnipotence. In an nondeterministic world, it just becomes more important to embrace it. To acknowledge that it is a fundamental part of not just the model, but the thing we're modeling. In this respect, removing a noise variable doesn't just result in overconfident results, but missing a fundamental variable. Of course, noise can also be used to capture unknowns. So in any case, the noise has the be embraced. We should reduce it as much as possible, but it can't ever be removed. That's the problem: people are removing noise from their models (e.g. using a benchmark to determine evaluation absolutely)
What policies are those? I thought 'publish or perish' was an expected outcome given the extreme supply/demand situation in the academic job market.
I don't think anyone would argue otherwise. The issue is that if there's a culture that encourages (incentivizes, not advocates) cheating, then we should expect a culture where more people cheat. I think right now it is hard to argue that the current culture doesn't encourage cheating, and if we're being honest, doesn't have a good means of identifying it (publish or perish also decreases reviewing quality).
There is utility in both speed and in a slow step. The truth is that we need both. A continuum of methods and ideas. After all, we are searching in the dark and our only real advantage is parallel Monte Carlo search. The problem is that this is incredibly difficult to accurately evaluate and we are arrogantly attempting to evaluate exceptionally noisy systems with extreme precision without even having a basic model of that system. We could relate this to businesses too, but the timetables for them are typically shorter and so the chaff can more clearly be gained (often post-explained through pure intent instead of it's combination with luck). Then again, we are trying to make academia a business, trying to apply their noisy successes to a completely different model.
Ugh.
There's plenty of honest researchers in the field.
You don't have to be corrupt to build a nice career and, now that I think of it, only people who are equally corrupt seem to embrace/justify/tolerate these practices.
This is multiplied several-fold when you are doing this populist intellectual dinner talk circuit in one of several countries whose elite are anxious to establish (or regain) their own scientific prestige on the world academic stage and will throw unbelievable amounts of money at the most brittle and flimsy proposals so long as there is an R1 institution attached.
Exactly. We are finally starting to see the unwinding of the 20th century scientist myth. With the advent of corporate culture in the academia, it now attracts and nurtures very different personality types.
The social sciences have a certain hypocrisy to them. "We do not need to be as grounded in math as the other sciences" runs up against "Our results are mathematically significant and deserve the same respect as the other sciences."
Doing statistics in the hard sciences is easy, yet we study them at a graduate level. Setting up variable controls in hard sciences is easier, yet we spend years training people on eliminating even the smallest chance of error through rigid lab protocols. The hard sciences allow you to collect enough data-points, that a large effect size is almost always statistically significant... yet we report p-values religiously every step of the way.
Social sciences have none of those affordances, yet they somehow get away with even less mathematical and statistical rigor. I mean this both in training and in practice. The social sciences are harder to control. It is a 'science' that can say very little about anything (purely due to difficulty of setting up controlled experiments). Yet, social science produces a disproportionate number of 'spicy' results.
Not all fields will produce 'ground breaking' work and that should be fine. I'm not sure where the incentive/impetus for it comes from, but social scientists build storied careers on results that would get rejected from any respected science journal for lack of rigor. It is as if there is this pressure on every academic to prove something fundamental about human nature few years, all while it's plainly clear that we as humans know almost nothing about human nature. If centuries of work in this field has not been able to find a few foundational facts (fancy alliteration), then maybe the field needs to reconsider its ambitiousness and self-assured confidence.
IMO, mathematical and computational “literacy” is particularly low in the medical and life sciences compared to the social sciences, physics and chemistry are not quite as bad.
Look at the most modern open source R and Python packages for statistical inference… probably 90 percent are developed and maintained by social science research labs
Bayesian statistics is great, you can get any posterior you want by choosing good priors.
"This study provides 5dB of evidence that the pigs do not reactionless flight capabilities, and 300dB of evidence that I don't know how physics works. However, according to my priors, the flying pigs hypothesis still comes out ahead." said no study ever.
It's best to model journals as magazines that publish anything that vaguely resembles science (has equations, charts, citations, claims of data). It doesn't have to actually be science. If you want to get a paper published it's totally normal to write 10,000 lines of C or FORTRAN, claim it models some natural process, publish a paper with the graphs it outputs and call it a day. The code isn't available. The input data isn't available. The assumptions aren't documented properly or at all. And the outputs are unvalidated, therefore not even laying the foundations of being evidence of anything. Doesn't matter, it'll get published, in some fields this is the vast majority of what gets published (e.g. epidemiology).
Personally, I always use a uniform prior unless I am able to cite or present the data that led to something else.
The best experimenters I know in terms of monitoring and controlling external variables are chemists, but sometimes they just use the wrong statistical test.
Biologists seem to generally be bad at both, and physicists seem to generally be good at both. I honestly think this is about the culture of the fields rather than knowledge - they all start as good scientists and just unlearn what they don't "need."
Biology is definitely a "real" science yet the problems with fraud in biology are, if anything, worse.
See the recent cases involving fabricated data in a Nature paper about Alzheimer's.
https://www.science.org/content/article/potential-fabricatio...
That actually seems to be a common pattern, there's another case now with strong evidence of data fraud in multiple Science/Nature papers, possibly implicating the president of Stanford.
https://stanforddaily.com/2023/04/25/stanford-president-dodg...
The problems seem less severe in physics, but there was that recent case where supposed experimental evidence of majorana fermions had to be retracted, with some research practices that look dubious in retrospect.
Or the famous Schön scandal at Bell Labs in semiconductor physics, where there was outright fraud.
Could it be there's more than one problem? :)
Social science is non-replicable because the parameters are too fuzzy. (People are more diverse than electrons).
Lab science is non-replicable because the experiments are too expensive to set up, and sometimes are intentionally kept as trade secrets for economic reasons.
The pressure in most of these sciences is to consistently get a paper for every expensive experiment that is done, but if you're doing that, and published work has to be a positive result, you're not really doing science. You could just publish your hypothesis and be done with it if you already know the result of the experiment will be ginned up in some way to be positive.
I don't see how we do this. Ultimately, the buck stops with the public. Governments fund the sciences because the public demands it. If scientists spend more time reporting negative or ambiguous results then the public begin to question what they're paying for.
People want breakthroughs and scientific revolutions, not "business as usual." And this bias is not limited to the public, scientists themselves want to see new science.
Is this a scientific fact? Secondly, is it an epistemically flawless fact (which is similar, but also different)?
In what way exactly does "the public" "demand" that which science does?
Why don’t you check on Google Scholar and report back to me?
In what way exactly does "the public" "demand" that which science does?
In the economic way that we say most things are demanded. I’m not saying that people are going to be out in the streets protesting if scientists start publishing more negative results. I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results.
Standard conventions for burden of proof I suppose. Surely you don't actually have no evidence to back up your claim, do you?
> In the economic way that we say most things are demanded.
Are you discerning support for science from economic data of some sort?
>I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results.
And might it be possible that the public's perception of science could be affected by all of the free marketing it's gotten in the last five or so years?
So I don't get how this supports the current framework.
This is basically just noting Goodhart's Law, or as I like to say "Warning." Evaluation is incredibly difficult and every metric can be hacked. Making "meritocracy" a fool's errand since a metric alone is not enough (it needs the associated uncertainty value, which is non-zero).
The problem here is that we're searching through an incredibly high dimensional non-convex solution space and assuming that a simple optimization function is a global optimizer.
Instead I'd just love to see academics pursuing their ideas and discussing them in the open. I don't think you'd have a significant problem getting people to generally align in a single direction, as currently happens, but I think it also allows for others to challenge those directions as well. Essentially research is performing some sampling in that search space but we're turning the learning rate way down. So we get plenty of works where novelty and incrementalism is indeterminable, due to the speed in which we must produce. Instead let the academics determine their own parameters and let time sort it out. Ensembling is a very useful technique, especially when uncertainty is high.
This just advocates for a system that more accurately captures uncertainty in the models. Which, would technically make the results replicable but reduce their impact. At least in the short term. Impact in the long term is harder to determine.
Though, I really think A LOT of fields need to be far more honest about their levels of uncertainty and any ones that aren't (especially empirically based ones) are bordering snake oil territory.
Social scientists tolerate their peers who see themselves as agents of change more than as scientists. A Stanford education specialist, who wrote California's new mathematics standards for public schools, framed--among other things--a well-reasoned takedown of one of her papers [1] as personal attacks [2].
Consider the same happening in physics. Paper published. Critique follows. Is there any world where a physicist maintains respectability by ignoring the content of the critique while claiming harassment?
Yet that's not only tolerated in many social sciences, it's the norm. Add to that lucrative consulting gigs, and researchers have a perverse incentive to (a) extrapolate N=2 studies of hung-over undergrads to corporate America and geopolitics and (b) personally attack anyone who dares challenge their premise.
[1] https://www.nonpartisaneducation.org/Review/Articles/v8n1.pd...
[2] https://joboaler.people.stanford.edu/sites/g/files/sbiybj286...
This isn't true at all. It's quite easy to create well-defined survey questions and experiments. It's entirely replicable.
The main problems with social sciences are that they have historically been perfectly fine publishing studies based on statistically tiny numbers of participants, as well as assuming findings from tiny groups such as university undergrad test subjects could be extrapolated to all of humanity.
These problems are eminently solvable with stricter statistical standards, and much greater cross-cultural research. It's just research becomes that much more expensive.
I don't know that anyone else does, but I like to draw a distinction between experimental scientific inquiry and rigorously observed phenomenology. Properly speaking I say a great deal of what's called science is really a species of phenomenology. For example cosmology is pure phenomenology. We can't actually run experiments on what happens when you smash two black holes together. However we can rigorously observe the phenomenon and abduce a descriptive model that can potentially even make predictions. But the word "science" has become a totem, so there's a lot of cargo culting[1] and punishing blasphemers.
However, I don't really think that whether a field is science, phenomenology, or some blend of both has much effect on the rate of fraud. Rather, I hypothesize the dominant factor in the incidence of fraud is the perceived magnitude of the potential benefit to be obtained. The Alzheimer's fraud is a good example of that. There are hundreds of billions and maybe even trillions of dollars in play in that space. Drug companies have already poured billions down the drain chasing dead ends so anyone who can sell them hope can potentially get a substantial payday. An amusing corollary of this is that given the relatively small stakes social science fraudsters are playing for, we can conclude that they are small-minded in the sense that they aren't thinking nearly as big as the drug company fraudsters.
[1] https://calteches.library.caltech.edu/51/2/CargoCult.htm
“It is difficult to get a man to understand something, when his salary depends on his not understanding it.” - Upton Sinclair
Except possibly even more high stakes than that, since academics' very reputations are on the line.
https://en.m.wikipedia.org/wiki/Hat_puzzle
Gödel's incompleteness theorem (philosophical); 'Theorem states that no consistent system of axioms can be listed by an effective procedure, capable of proving all truth' (-;
https://en.m.wikipedia.org/wiki/Epistemic_logic
Hint: HN in 2016
regards...
Not to mention the whole tenure system is questionable. Yes, it's good that at least a few percent of academia has job security, but the selection process is full of perverse incentives and arguably society could get better results if tenure was for 10 years and based on a lottery.
Or we could use fancy Latin suffixes:
sociology, psychology, economicology, biology, chemiology, pharmacology, mathology (https://www.youtube.com/channel/UC1_uAIS3r8Vu6JjXWvastJg).
Wasn't it called social studies? When did social studies get branded as science in the first place?
Of course the best irony is the current fad of rejecting "objective truth" (on whatever grounds are useful - it's racist, etc) and yet still claiming you are doing a science. Not the search for truth but rather the search for your truth.
https://www.cambridge.org/core/journals/modern-intellectual-...
The best psychology work tends to be much closer to philosophy than science. But philosophy generally doesn't get much funding.
Anyone interested in psychology will ultimately be forced to optimize for funding, which inevitably leads to fraudulent work since psychology based on scientific principles simply isn't possible yet as we don't know nearly enough about the function of the mind to explore the areas psychology is interested in.
A good social scientist understands everything you're pointing out. This isn't news to anyone nor is it even really controversial.
But we can witness phenomena in society so people try and understand them. Would you prefer we not try and understand them? Just give up because it's too hard to figure out?
Stop ignoring the important nuances of the different problem spaces science works in, and lacks adequate sophistication in its methodologies for dealing with those nuances resulting in these gong shows regularly happening and then explained away as if there's no fundamental problem.
Some are purely on the popular science side of failure. The 10k of practice was basically thrown out, but the original research still seems good. They never claimed "if you do 10k of work, you will be good."
The "marshmallow test," though, seems completely tossed? Maybe there was something there?
Anchoring and other items? Not sure how well those have survived. :(
https://en.wikipedia.org/wiki/Replication_crisis looks to be a good article, though I haven't finished reading it.
- Stanford Prison Experiment; Not an experiment, abuse was scripted, experimenter constantly intervened, reactions of participants were faked, and there was not even a scientific hypothesis they were testing. Definitely read the paper[0] debunking it.
- Milgram Experiment: (the one where people were ordered to shock actors) No good evidence for it. Researchers did not follow script, implausible levels of agreement between different experiments. Killer line is, “only half of the people who undertook the experiment fully believed it was real and of those, 66% disobeyed the experimenter.”
- Robber's Cave: (the one where two groups of kids immediately formed tribal hatred between one another) The conflict was orchestrated by experimenters and the experiment was actually repeated because the first time the kids absolutely refused to turn on one another. More information at [1].
- At best, weak evidence for implicit bias testing and stereotype threat.
- Weak evidence of "facial feedback" (smiling causes a good mood and frowning causes a bad mood)
- Good evidence against "ego depletion" (the idea that willpower is limited in a muscle-like fashion)
- Mixed evidence for Dunning-Kruger effect
- Questionable evidence for "hungry judge" effect (the idea that judicial sentences are massively more merciful in the morning and after a lunch recess due to "ego depletion" - this is also thoroughly debunked here [2])
- The 10,000 hours of practice leading to expertise idea has been disowned by its proponents
- No good evidence that tailoring teaching to students’ preferred learning styles has any effect on objective measures of attainment.
- No good evidence that brains contain one mind per hemisphere. i.e. the left-brain, right-brain split that people talk about, especially after the link between the hemispheres is severed.
- No good evidence for left/right hemisphere dominance correlating with personality differences.
Per https://danluu.com/dunning-kruger/:
- Most people talking about Dunning-Kruger have no idea what it actually means. The actual purported bias is much weaker than people claim - basically, that everyone either overestimates their ability, but that estimation is still positively correlated with actual ability, or estimated ability has basically no correlation with actual ability and everyone is just guessing.
- Increasing your wealth does in fact make you happier at a predictable rate. There is no "plateau" of wealth or income - what appears to be a plateau is misleading displays of data. In effect, increasing income by a proportional rate will increase reported happiness by a fixed rate. For example, say you make $10, and your happiness is 50. Then, your income increases to $20 and your happiness increases to 60. Then, doubling your happiness is necessary to increase your happiness by 10. It's a logarithmic function; plotted on a standard axis, it looks like a plateau, but plotted on a logarithmic scale and it's a constantly increasing line.
- Hedonic Adaptation (aka the hedonic treadmill) is a myth. Bad life events (divorce, disability, death of a loved one) all have negative long-term effects on happiness. Vice versa for positive events.
[0]: https://www.gwern.net/docs/psychology/2019-letexier.pdf
[1]: https://www.theguardian.com/science/2018/apr/16/a-real-life-...
[2]: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...
https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Sokal_affair
Some of it is used in policy. So when some convicted murderer is out on furlough and kills again, am I wrong to wonder if that was some of the top-notch applied psychology at work?
The work here isn't fraudulent because of a mistake or because the author has shaky statistics knowledge, but because it was fabricated.
My experience in CS is that the replicability of experimental results is embarrassingly bad, but this isn't making headlines in the same way so people don't consider it to be a problem.
This is data fraud in psych. Data fraud has also happened in plenty of other fields. When it occurs in these fields it is seen as a one-off. When it occurs in psych, it is because the entire field is useless garbage. That's not the lesson to take away here.
I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.
Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.
Outside the high-profile cases it seems accepted norm that papers perform far worse when scored against somebody else's benchmark. The real measure of quality is how big the gap is.
Adding to complication in the psych field is the widely observed phenomenon that it draws students with psych problems. Like those who would be willing to fabricate data, for example.
But hey, at least it isn't sociology.
There are resilient exceptions in academia in general, but soft science fields have led the way in the decay of academic standards.
There are dozens of well known fields that are built on epistemological quicksand yet which point-blank refuse to admit to or talk about their problems. Because they have a culture of deny deny deny, even when the evidence is overwhelming, they actually get less attention because why bother doing a nice writeup of fraud if you know the result will just be stonewalling? It's better to focus on fields where there might be some actual response, a chance of improvement, no matter how minor.
Some results do replicate. Big Five personality traits, general classification of mental illness, mainstream IQ results, and human perception and performance. All psychology is not false and there are interesting things to learn from the field.
But the severity and frequency of fraud and non replication is worst in psychology. So much so that half of what you learned in Psych 101 ten years ago does not replicate.
The big challenge for the field is that humans know a lot of psychology. We can track our own thoughts and we are constantly interacting with people and trying to understand their psychology. A biologist who studies ants intensely for 5 years is maybe only one of 100 people have ever watched them that carefully. They'll find lots of new stuff.
Psychology doesn't have powerful techniques than that for determining new truths about humans. It's still mostly give a questionnaire or put people in weird situations.
So there are a lot of researchers hunting around for original ideas that are undiscoverable with our current tech. Some of them are bound to give into the temptation to just make up an interesting result with manufactured data. With p hacking they might not even be committing fraud, they are desperate for a positive result, when the get one they stop looking and publish.
So I would still guess in 2023 that about 80% of "new" discoveries published in journals won't replicate. That rises to 95% with journal publications that are published as university pr and reported in the media.
For coverage of what to remember to unlearn Rolf Degen is great: https://twitter.com/DegenRolf
Popular examples that do not replicate (mostly from Rolf):
Repressed childhood memories Social media harms Search bubbles and echo chambers Women prefer masculine men when fertile Priming (you see a fight in the hallway and then don't cooperate later) Power posing (puff out your chest to feel more confident) Watching eyes make people more honest (in Kahneman and Malcolm Gladwell)
Notice that a lot of these are interesting, and you want them to be true. That's not enough to make it so.
As a bonus, an interesting interview with Daniel Kahneman: https://www.edge.org/adversarial-collaboration-daniel-kahnem...
* Plenty of studies inherently cannot survive 20 years. Send out a survey and ask people "on a scale of 1 to 10 how happy are you". These results will not match the results in 21 years simply because the population you are studying is different. This example is simplified, but explains why many results change over time.
* Plenty of studies have survived 20 years, and those that study psychology are really clear about which knowledge is foundational and which is on more questionable ground. The Big Five personalty traits is about 40 years old and has been found to be consistent across cultures. A ton of psychological research around psychological responses has survived the test of time.
* Psychologist often relay on more questionable data than other scientists because it's much cheaper and easier for psychologist to collect questionable data than other sciences. This allows for a wider exploration and makes research much more accessible. Anyone reading a paper with this type of data will be naturally skeptical. I am specifically referring to data collected from self-reported questionnaires, often collecting a sample that generalizes poorly over interesting populations.
But that's not the only problem. The problem is that people try it anyway. They have some hypothesis within a theoretical framework which is embedded in other frameworks, none of which is proven. How could they be? But, they set up an experiment anyway, and (usually after a couple of attempts) they find something that can reject H0. Then they publish an article stating that their theoretical framework is a fact.
There is so much wrong in this process, even ignoring the manipulation of the stimuli and conditions, and the statistical procedures, yet the theory has a good chance to make school and become the subject of research in dozens of psych departments, each contributing articles to the literature about it. After a while, an essentially flawed theory enters the handbooks.
Then, if you want to make a name for yourself as a researcher, and you need publications, you can simply attack older theories. They'll crumble like cookies, you get your publications, and the cycle restarts.
And as critical as I am of psychology, other social sciences have even lower empirical standards. Educational sciences, sociology, linguistics all have very little to show for a century of research. And that's ignoring disciplines like political sciences or history.
Former semiconductor researcher here. It is quite common for researchers not to believe published papers - even in prestigious journals from well known researchers. No one bothers replicating, and they don't put enough information in the paper to replicate (competition - they don't want others to get the secret sauce).
This was true for both computational and experimental papers.
Oh, and I did know one person personally who falsified data (and was caught). He just transferred to another prestigious school and got his PhD there instead.
It was quite demoralizing and helped me decide not to pursue academia.
The scary part comes about 50 years ago when everyone woke up one morning and figured out that the "happens to include science" part wasn't strictly necessary.
"Oh no! Anyways."
When most people are stuck in a rat race to get rich and "become successful", there's not much energy or reason to care about some drama, unless it's entertaining or political, perhaps. If my employer follows the newest workplace fads based on some studies, I might care. Otherwise, why should I spend mental energy (at the very least) trying to get the messes of scientific research cleaned up?
People are morons, generation after generation, what else do you expect?
Imagine spending time to prepare a paper then collecting data just to see it doesn't show what you hoped it shows. Your choices are to bin several weeks/months of you work to the detriment of your career or help a bit so the data behaves. The risk is close to 0, the reward (or lack of punishment) is clear. It's crazy to not expect significant % of individuals facing the choice going with the massaging the data option.
It all flies in social science because most of the stuff they "research" range from interesting but not very useful curiosities to completely pointless. No devices are being built based on it. No serious investment being made. No policies are going to be affected or if they are then studies are picked for ideological compliance not for describing reality well. You're also not going to face a lawsuit as you might in say medicine. "Your honour, I've read the study , started breaking the rules and got expelled from school" is not going to get you far.
there is a large amount of fraud (real fraud, photoshopping images to show fake scientific results) in biology, some of which has directly led to serious pharmaceutical investments in drugs that don't work. Recently Cassava Sciences got in trouble for this. https://www.nytimes.com/2022/04/18/health/alzheimers-cassava...
as far as the alleged data fraud in this story, at least two of the authors have made a lot of money giving talks, consulting, and acting as scientific advisors on the basis of their research, so there's also fairly serious financial stakes.
People who don't need guidelines gain nothing from them. People who need them won't sign up to them. Meanwhile I have to fill in a form and have a 2h meeting before I can shine a laser on an (already dead, commercially available) butterfly's wing?!
I want to be clear that I would phrase that as “based on consensus,” and not as bullshit. I have seen so many doomed/failed projects/models that have come up because people say “well, it’s math, so it has to be right.”
Sorry if I'm missing your point?
A ethicist will never be caught using real inputs at all, precise or not.
A Psychologist will tell you 86% of people taking treatment X recovered. But miss out that 85.8% of people on placebo also recovered. Those numbers will be from a study of 10 people.
Mathiness is a term coined by Nobel prize winner economist Paul Romer to label a specific misuse of mathematics in economic analyses. An author committed to the norms of science should use mathematical reasoning to clarify their analyses. By contrast, "mathiness" is not intended to clarify, but instead to mislead. According to Romer, some researchers use unrealistic assumptions and strained interpretations of their results in order to push an ideological agenda, and use a smokescreen of fancy mathematics to disguise their intentions.
Your model can be completely internally consistent and allow you to study things in the context of your assumptions perfectly, but that doesn’t mean it doesn’t have critical flaws that disqualify any learning about the system you’ve modeled.
The lack of understanding about what implications a representational choice have for the kinds of relationships a mathematical model can describe, and the lack of understanding concerning what are and aren’t critical domain components to model that’s difficult.
People often forget that there are two translation steps that have to take place. Domain -> Math -> Domain. That second one is too often implicit or assumed to follow, especially in ML.
I didn't say Humanities were worthless. I just said there are no provable results there. "Is X ethical?" - no one can prove it but we will discuss it as long as someone will pay us to. Since there are no firm truths, anyone can get a degree (or more) without actually gaining the skills you refer to. In fact actually gaining the skills you allude to makes someone LESS likely to get funded or famous. It is extremists filled with certainty who rule these subjects...
that's bullshit. Classic liar's paradox and the solution is that sentence is ungrammatical in a strong sense of universal grammar ... which I can't prove, and I'm not philosopher enough to concern myself with lists of formal fallacies to make up for it
Remember, not an opinion or an argument, a cold hard, proof no one can argue with.
Also, science doesn’t really prove things true. It can prove things likely. And it’s really good at showing what is not true. I think in general humanities fail to address falsifiable things. This is why we have a reproducibility problem and why it’s mainly bullshit.
'M' is also largely bullshit. Worse, it's got immediate consequences.
https://trends.google.com/trends/explore?date=all&q=%2Fm%2F0...
Of course if “these kind of studies” refers to fraudulent ones, then yeah no one would disagree.
Like many things, research is often spoiled by human nature. Researchers want to prove their hypothesis so desperately that they’re willing to do whatever it takes to make that hypothesis true. Perverse incentives are everywhere in the research world.
How about we stop trying to force “policies” on a population who doesn’t care about them and using unscientific studies to convince them they need it. I’m sure you’ll have a snarky remark to this, in which case don’t bother.
None of these is actually a viable approach to, well, anything.
If you are genuinely confused about my reply, I suggest you reread the post you responded to and then reread your own post, focusing on what you were saying in the context of the post you responded to.
Lots of people working in policy don't have any real ethical or political engagement with the problems they are supposed to be solving. They lack vision and purpose. They choose to believe that irrelevant and trivial interventions can make huge inroads into difficult social problems, because they are like the man looking for his wallet under the lamppost. If making progress on really difficult problems requires deep thought, courageous leadership and persistent hard work, then they aren't going to be able to solve them anyway.
The social sciences are largely irreproducable, and our socities tolerate it because they give the bureaucracy the tools it needs to manage the legislative branches in a form of capture. Show me a study that demonstrates an official narrative is baseless, and I'll show you "a problematic fringe theory from a now-former academic."
The lede of the article also calls it "accused of".
[1]: https://datacolada.org/109 [2]: https://datacolada.org/110 [3]: https://datacolada.org/111
The difference of "allegedly" is big to a publisher (missing it can break their own company), and to a journalist.
Because of this, I think it's also a big difference to the savvy reader. When an outlet dispenses with the "allegedly" or phrasing as someone else's claim/assertion, that feels to me like a rare occasion on which they really want to impress that it's definitive. And that they know what they're doing, and that experienced readers can tell. Never crying wolf. (A little like how you'd say "literally" once in a decade or lifetime, with emphasis, for impact, when it was really true.)
I imagine this stuff happens at state schools too, but yet another nail in the coffin of their reputation.
Sincerely, a person who went to a school you've never heard of, but makes a ton of money because I know 4 skills that synergize and scarce.
If scientists had more than a citation index to establish productivity so that they can justify getting or keeping funding, the community could probably make some headway towards addressing behavior such as Gino's. Outright fraud is less prevalent than bad science or misinterpreted or misunderstood results. Personally I am in favor of a replication index for all original work, at least.
Often for business school profs, their work will be judged less on academic acceptance (peer-reviewed publication, etc) and more on its 'real-world impact'. You are seen as a good academic of marketing if you can market yourself into Ted talks, pop science publications, and ultimately, influence on 'decision-makers' (the sort of politicians and thinktanks who believe that eg healthcare can be solved by painting hospitals orange).
So there are strong incentives to falsify results but they are quite different from those which bear on actual scientists.
Hey, good for you!
I always find it odd when people choose this as a metric for success. Why not "a person who went to a school you've never heard of, but has a job they enjoy deeply and that gives them purpose"?
Similar to why so many people want to get Google, Meta, Microsoft, Apple, etc on their resume. While there are a ton of amazing small companies they just do not have the same brand recongition.
I think the economic term for this is "signalling." Having associating with these brand name schools gives you instant credibility: https://en.wikipedia.org/wiki/Signalling_(economics)
It's not odd. If your metric was "be as useful as possible to others" then making lots of money is an excellent proxy metric.
I disagree that the two types of people are "rich people" and "people with no money"; the latter of which is vanishingly rare in developed countries.
> You can make money convincing rich people you're useful to them when you're not.
You can, but I don't think this accounts for all or most or much movement of money.
You may live in a bubble
Also, financial success is on-topic here, as this forum originated for Silicon Valley bros seeking wealth. In past lives, we were "greed is good" Wall Street bros. I'm sure some percentage of our earlier counterparts started out really loving spreadsheets, before the schemes.
Because when a person has a lot of money, it vastly increases the scope of available activities that they enjoy and give them purpose.
Because by and large people go to university to have a better chance of making more money
Most aspirational people who are out to make mega bucks are trying to put distance between themselves and the time they didn't have that, and to setup the structures in their life so if they have kids, they can give that easy going life to their heirs.
I always thought people in academia don't really make much money compared with what they could make in business. Adding to it that it takes a very long time and lots of hard work to get "to the top - get tenure" I'm amazed we're still seeing any scientific progress being made.
Those would be? (Asking sincerely.)
Most scientific studies are plain up boring and don't find anything new.
Maybe papers should be graded on "how well was this study executed" instead of "what kind of unexpected result did we end up with". Because the second is really asking for exaggeration, lies and fraud.
Note the selection bias: "High-profile." Plenty of grad students still do honest research. They just don't generate thousands of citations, media coverage, or find academic jobs.
I firmly believe tenure should be obsoleted, and academics should be fired for any sort of documented dishonesty, anywhere. Lying is not okay if you're engaged in the search for knowledge. If a person intentionally lies in a popular science book, a court case, or a newspaper article, they should be out.
but no, we have to “trust the science”
for example, if you create a study whose results show that diversity is good in some way, you’ll get endless citations and orgs will make policies after it. even if it’s fraudulent, no one’s likely to look too deep into it. you wouldn’t be able to publish one that said the opposite. or if you did, you’d likely end your career
Isn’t that self-fulfilling and as deconstructional as can be.
The one thing I wont bet on, is people suddenly deciding to accept that they are the genesis of their misery and their joy, and stop seeking an outside "other" source of authority.
> by Uri, Joe, & Leif
When publicly laying out a case like this, which could destroy the reputation of someone they identify by full name, I think it'd be good form to put one's own full names on the argument.
(Of course they'd have to stand behind it if they are sued for defamation, and the full names can be found elsewhere on the Web site. But that's not my point.)
Putting full names on the piece that people will read says to the reader that they take it seriously enough to put their own names on it.
And the whole alleged scandal is about integrity and what you'll put your name on.
They're not hiding it to any extent. It's a big blog that has hundreds of posts. Many blogs refer to authors by their first name if they expect their readers to know them, or easily find them.
> ([...] and the full names can be found elsewhere on the Web site. But that's not my point.)
These are meaningless weasel words. It sounds like you wrote the post first, then belatedly realized the authors aren't hiding their names at all, but wanted to keep your post.
Those are not weasel words. I said that in my original message. I anticipated your argument and addressed it, before you made it.