Noted study in psychology fails to replicate, crumbles with evidence of fraud
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology.
Reproducibility crisis is real and it sucks for all the people that read individual books and don't know about reproducibility. I feel people that write those books should also put themself into higher standards and stop writing narrative books based on individual nonreplicated studies.
You can read about reproducibility crisis here: https://www.theatlantic.com/science/archive/2018/11/psycholo...
This is about the latter. This is the kind of corruption that spreads. It's a very bad thing.
It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well.
Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something.
Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc.
Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics?
A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.
It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve changed a parameter that happens to be important: now find it!
I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before.
Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place.
Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.
I think framing it in this lens is helpful because it points toward a solution: capturing more of the parameters more explicitly and varying them more deliberately. If we just say, “oh x y z studies aren’t replicable, field [x y z] is bullshit,” we are ending our inquiry prematurely. We are ending it at the falsification of Newton’s initial theory and saying that because leaves fall at a different rate, that gravity doesn’t exist.
You're not going to find the parameter which makes a high school nutrition program deterministic. It shouldn't be the goal.
If you can’t find the parameters that make it reproducible, what value does it have in the first place?
Surely there is one, but it's maddeningly hard to pin down. Especially when the experiments are hard to perform and control properly, which applies both to psychology and astronomy.
Remember that scientists existed before philosophy of science, including formalisations of The Scientific Method. Darwin wasn't following the scientific method, the method followed him.
If you can't control parameters so that the results you are claiming are reproducible you are just publishing noise ?
Should a journal paper never be published until the field has established all the fundamental properties? How could it discover those fundamental properties without the publications?
We found that students participating in the "2hr active recess program" have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years.
It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities.
Basically, the world is not a laboratory. You're not predicting the trajectory of a projectile, where unknowables can be quantified as a function. It's all 3 body problems all the time.
Yet, we as humans do manage to make sense of the world. We run schools, companies. We learn more about how to do those things. Formal studies are an important part of learning about how to do those things. Scientific mindset and methodologies are useful to this. It's a lot better than just philosophizing the whole thing. The type of knowledge we gain, OTOH, isn't like F=MA. It never will be.
The value of the above study does relate/imply to it containing knowledge. It's a partial knowledge of a general problem. It is not complete knowledge of a particular problem, like you would get in a lab. Neither lab experiments or wild studies tend to replicate in one another's world.
No matter how good the study, how many times it replicated, or how well it worked throughout the 80s and 90s... there's no guarantee that your program is working well or a guaranteed explanation for why it isn't.
Elucidating confounding variables and controlling for them is the entire fucking job of a scientist.
To your example, if they publish a paper that explicitly highlights that it worked well for sedentary kids with typical diets of X, Y, and Z in region Q, then it failing to be a found in other environments isn’t a problem.
It’s a problem when these clowns do a test on one specific group of 20 people and then generalize to, “doing X leads to Y in humans”.
The answer is absolutely not loosening up the constraints. It’s to keep lambasting people who generalize before anything has been reproduced widely.
Popper’s science is fine. Morons who don’t realize what they measured are the problem.
Many times the first or first few teams won’t know what the confounding variables are and subsequent research is needed to find them. They should still report their initial “hmm, that’s novel” findings even if it later turns out that it’s explained by confounding variables.
*-No true Scotsman
No. This is methodology. The job of a scientist is to advance scientific knowledge. This is as vague as it needs to be.
Popper didn't do science. He did philosophy. He also did a fine job of formalising methods, and abstracting the methods scientists were using into a Scientific Method. This scientific method is a great achievement and a wonderful, multipurpose probe. That does not mean it fits in all orifices. The job of a scientist is not to continue hammering until the probe fits.
I think it has to to be held out as a possibility that there is actually no science behind how we run schools and companies. One way this could happen is if non-scientific ideas simply overwhelm the ones that are grounded in good science.
The fact that it isn't like F=MA also means it will never advance beyond producing collections of amusing factoids that are unrelated to one another and potentially false.
However, a big chunk of this statistic is not actually fraud. It's just false empiricism. If you look at a lot of the actual papers, I think you'll find that a lot of them just aren't trying to do empiricism... at least not strict empiricism. They're forced into that structure by a convention.
Either (1) "Not Science" becomes OK and extends to economics, some areas of ecology, and such. (2) The Scientific Method gets renamed "Method A," and the concept of scientific pursuit is more flexible, depending on what you are studying or (3) We force these poor bastards doing economics, psychology and such to pretend that they're doing a method that they're not... or publish in the prestigious "Unscientific Economics & Psychology" journal.
> Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’
Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application".
This happens in software as well. If my system works on my specific laptop but anyone with a slightly different version of the programming language, imported libraries, or operating system will be unlikely to get it to work, then it's just not good software. The solution is not (usually) to put something in the README that says "will only work with python 3.6.7 on Ubuntu 18.04", the solution is to change the software so it is more generally applicable. And it's on me the programmer, to make that happen. If the scientific phenomenon is specific to 1990's Tokyo but not 2000s Miami, then some major variable is being left out, and the science requires finding out what that is.
I agree with your basic point that there is a big difference between non-replicable and fraud, though.
Maybe not technically, but it does imply that "this specific situation" should be defined specifically, hence controls. I agree that broad applicability, and replicability are strong indicators of truth value. That said, there are pursuits of rational knowledge outside of Popperian ideals. That's not represented in the way Science, as an institutions works.
Sometimes things are even more complicated than you already believe them to be. And it may take more than just a replication study to say a study is wrong or fraudulent.
If it doesn’t replicate, it should be relegated to “less than observational” level evidence. Why would you believe that the assumed intervention had the claimed effect, if it doesn’t have it when you try again?
If it fails to replicate certain statistical outcomes, things are more complicated.
Look at emotional intelligence (EQ) as an example. One of the early books on the subject written by a journalist which was based on a shaky foundation and then added a bunch more sugar. It might as well be a self-help movement which could spawn coaches, authors courses and certifications.
Even if Ariely had no role in the creation of the fake data, it seems hard to defend his behavior on NPR: hyping a study that he knew to be extremely problematic.
Journals would then have to select studies based on their design and based on the question and scientists could just do science.
Edited to add (from Wikipedia):
> Over 200 journals offer a registered reports option (Centre for Open Science, 2019),[33] and the number of journals that are adopting registered reports is approximately doubling each year (Chambers et al., 2019).[34]
> Nature Human Behaviour has adopted the registered report format, as it “shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.[36]
> European Journal of Personality defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”[37]
I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to fake the data without it being obvious. Once we have enough confidence in that, let's do preregistration, sure. And then negative results should be as incentivized as positive ones, absolutely.
I worry about the order in which we want to do these things. If people are faking data, we need to make this specific step harder to hide from reviewers.
If you incentivize people to fail to replicate each other without solving the lag between fraudulent data and retractation, you will get fraudulent negative results.
That could cause a lot of confusion. If the replication crisis itself starts to be fraudulent, scientific credibility is further damaged, negative results will be less credible, and fraudsters will profit from the confusion.
Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable:
You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sending 'bills' and the wares if, or if not been paid - 'you've been paid'. Viewed by the company-manager your position may a) be valid oder come b) 'overvalued' (...)
Stucked in this example, you talked to other 'employeees' during break and 'there was a time meeting on weekends', so you talked about work and there was a (special) kind of 'more transparency' than without peer review. ^^
Regards, hope this helps... (-:
I'm only trying to make a point about the order in which we should shift incentives vs increase fraud-resistance requirements for experimental protocols.
I think the latter is more important, and should be done first.
The journal that publishes failed science would be the biggest and most important scientific journal IMO.
The only concern I have around a disincentive to not publishing is that in some fields, the peer review process can be quite flawed and effectively hold back worthwhile publications for various reasons - sometimes there's too much of an expectation of specific ingrained methodologies, or an expectation of certain "doctrines" to be adhered to.
There can also be some significant gaps between disciplines, where neither discipline wants to publish the work, as they both feel it belongs in the other discipline, which effectively leaves only the more generic (and usually less prestigious) "any inter-disciplinary research" journals.
That becomes a problem if institutions are focused on publication metrics and venues as a priority, as it means the institutional incentives drive people towards or away from these kinds of work. Ideally we need negative results published in the same venues as positive results (in my view), so those doing this valuable work don't get "relegated" to the negative results journals, which will no doubt get less attention at tenure review time, or from grant funders looking at publications etc.
One important task of the advisor is to filter bad projects. Does the paper pass the smell test? Are the authors reliable or they had published "dubious" results before?
[I was recently discussing a paper in a virtual meeting. My conclusion is that Zoom does not have enough negative reactions.]
Also, before assigning an experiment to a graduate student, try to have something like a plan:
1) This will surely work and is interesting enough for the dissertation and to be publish.
2) This part is dubious, because it's related to a too resent research paper or there may be some hidden technical problems that are difficult to see.
2') An underspecified alternative to 2, just in case.
3) Extra task for bonus points and perhaps a Nobel price.
This doesn't always work but having the capacity to disbar lawyers for, say, embezzling trust accounts or remove a CPA's license and put them in prison for falsifying records, it certainly has a deterrent effect on similar crimes of trust.
How much worse would white collar crime be if everything was on the 'honor system' and there was no prison time for stealing from customers/clients/shareholders? That's what we have in academia: crime won, so now syndicates of crooks run the system. With other abuses of trust the material consequences are more readily apparent: someone's bank account is empty that should not be. With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base. It's more of a 'reign of error' than a reign of terror.
It corrodes trust in our scientific institutions. When that pop something talk by an acclaimed psychologist turns out to be not only bullshit but outright fraud, it’s not a huge leap to project that mistrust onto e.g. the medical establishment or climate scientists.
If anyone is privy to a platforms that would allow me to share proprietary data such that skeptics can run aggregated statistical analyses whilst limiting the ability to surface results with insufficient samples (to prevent scrapping the the proprietary data—if not proprietary then there is no excuse for not fully sharing) please comment!
If not, YC hopefuls here is your chance to ACTUALLY make the world a better place!!! Happy to use my platform to recruit as many social scientists as you need to secure funding for a POC—seriously, ping me!
Must haves:
* Easily palatable settings for data obfuscation (non-tech corporate GC must be able to grok the level of obfuscation)
* Trivial sliders to change level of obfuscation (eg min number of observation to run a “test”)
* Messaging system to facilitate requests and permission granting between authors and skeptics
* Enterprise-level data retention and documentation, such that changes to the “official” data set area reflected publicly
* Cloud based R, Python, Stata functionality (else the solution would have to run locally in such a way that the granular data cannot be accessed directly on local machine—and this would likely be a non-starter for corporate GC’s). Ideally, one could reuse their local license for proprietary software (eg Stata), if not then said vendors will delight in an additional “attach” license to charge researchers even more (see below, there are beacoup bucks that will gladly be spent on this)
Business model: researcher’s academic institution would (gladly) pay (if you get pushback, call the university’s academic HR department, find out who is in charge (Ie associate provost of research or HR) and email them asking they are prepared we’re to shine a spotlight on their own “Andrew Bird and Stephen Karolyi” like scholars). They will call you in 5 min.
GitHub will acquire you post-POC upon enlisting 2-3 major universities (again, I know what higher-Ed procurement cycle is like, contracts will be signed record time). If GitHub fails, Google scholar will acquire you.
Ideally planning data production, execution of data production and analysis would all be done by different people and published separately. At the very least you'd want total transparency at each stage.
Also not to argue semantics but when you hear the word "scientist" in common parlance it almost exclusively refers to people doing that kind of research. The "sciences" that don't have these problems tend to not be too big on using the word "science".
The reason why the social sciences (social psychology, criminology, economics, advertising/marketing) are hit hardest by the replication crisis is because they were not actually sciences! They are ways for our society to police its members' behavior (relationships, crime, money, attention) and they pretend to be science in order to gain prestige.
There's also, finally, the matter of scale and relevance. When a chemist reports that a rare chemical reaction doesn't perform in the lab as expected, then this is a failed replication; however, chemistry labs regularly replicate entire textbooks' worth of material for entire classes of undergraduates. Meanwhile, some of the most important claims in psychology, like priming, are failing to replicate.
Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution?
No unfortunately it is not. It's perfectly plausible to have a normal distribution with mean = 26k and sd = 12k. Although those numbers do look kinda weird, but nothing you could verify. I am not sure what the F(*) means here, maybe an F statistic? But that seems wrong if you are comparing two normally distributed samples, you might expect a T statistic.
To verify the distribution you would need a histogram or you could get fancy with a qqplot. You could also try a statistical test for a normal distribution but these fail on large sample sizes so the visual plots are your best bet.
What am I missing? The mean of a normal distribution can be any value. A normal with mean 2 and deviation 1 has deviation equal to half of the mean.
Is this related to the fact that miles driven can't be negative (unlike a sample from a genuine normal)?
> Specifically, we ran analyses examining whether the two conditions differed in the number of instances wherein reported odometer mileages ended with 0, 5, 00, 50, 000, or 500. Numbers that end with these digits indicate a higher likelihood that customers simply estimated their mileage. We detected no statistically significant differences between our two conditions in the instances in which these endings appeared (pooled measure: treatment, 19.9% vs. control, 20.8%; χ2 = 2.5, P = 0.12).
1. What does this mean? (“Pooled measure”) Wouldn't a mileage ending in 500 also end in 0 and 00?
2. If the numbers are uniformly randomly generated, shouldn’t the odds of ending in 0 be 10%, not 20%? Or is this an equal mix of fraudulent and real data in both arms?
Just curious because this is the one of the only bits of data for the fraudulent study in the 2012 article.
Did they get anything more than a slap on the wrist? Are they still selling books based on the concept?
The breathless commentary regarding how perfidious psychology is threatens to sprain the muscles that govern the rolling of my eyes.
The other issue is that a very niche bet/topic might only have a small sample of analysts placing bets. This would mean you are subject to increased risk of bias from a single bad analyst. Or simply large variance due to small sample size.
How would that affect the field?
I really wish this weren't the case, but "choice" isn't a thing - as far as anyone can tell. All the complexity on display in the world appears to be a result of feedback loops, it really doesn't take much to generate a continuous low interval stream of pseudo random feedback - predictable after enough observation. People who regularly purchase chunky peanut butter and cat litter have a significantly higher probability of driving a VW bug. That isn't the result of limited alternatives.
https://en.wikipedia.org/wiki/Linear-feedback_shift_register
However, I want to point out that "not replicating" doesn't necessarily equate to bad science (or, more accurately, bad practice on the part of the principal investigators irt accepted best practices in their field).
There are a bunch of studies in the social sciences that failed to replicate simply because a p value was slightly below a threshold target, even though the coefficient direction/magnitude &c... agreed with the earlier study.
That said, I'm not defending - at all - the paper discussed in the article.
Edit: Also [0] was on HN 5 days ago (discussion here: [1]). This is another blogpost that discusses [0]
[0]https://datacolada.org/98 [1]https://news.ycombinator.com/item?id=28210642
If they're replicating with a follow-on study of some kind, it depends on any number of factors, since it's almost impossible to do a follow-on survey-based or lab experiment or use a similar or newer dataset while accounting for and controlling exactly the same set of conditions (this is one of the issues that I've heard used to distinguish between hard and soft sciences).
E.g., if paper A has an observation count of 200 (insert your preferred number), a replication study could hand out exactly the same survey to another 200 respondents and get similar results that, nonetheless, fail to exactly replicate the earlier findings (or, over a small number of samples, fail to conform to a normal distribution).
This doesn't mean that the original study was "bad science" or that the social sciences in general are "bad", only that we have to temper our expectations and maintain a healthy dose of skepticism.
In order to determine what proportion of studies would fail to reproduce, you would also need to know the number of correct hypotheses in the first place. If 100 correct hypotheses and 900 incorrect hypotheses are tested, your sample of "p<0.05" results would contain 95 correct hypotheses and 45 incorrect hypotheses, so about 1/3 of the results would fail to replicate.
This is why there is such a large push for prepublishing of analysis methods, hypotheses, and experimental design, and later publishing of negative results. The initial distribution of hypotheses is important to know how to interpret the resulting body of work.
Suppose your testing a weight loss drug and in the first experiment subjects lost 5 pounds and the second they only lost 3. On the surface the first study replicates fine, except the second study has a weaker signal so it might not past the statistical threshold chosen before the study. This is fairly common issue as you should expect random noise to amplify a signal in about half of experiments and more than half of all published research.
Also, I don’t think you understand that there is a difference between “replicate” and “reproduce” as those terms are typically used in social and biological science. Reproduction refers to the methods and is simply conducting the experiment again, regardless of the outcome. An experiment should be reproducible, of course—if it isn’t, the paper was poorly written (the experiment might still be ok though).
Replication refers to the results: do we get the same answers as the original experiment? Bad science is one way an experiment might not replicate but other reasons are 1. Just a regular type 1 error in the original study, 2. A type 2 error in the new one, 3. an overlooked methodological difference or mistake somewhere. No. 3 doesn’t mean bad science, it could be just normal scientists making normal mistakes, as they sometimes do.
Also, you don’t “hypothesize” an experiment. You come up with a hypothesis then test it with an experiment. (Sometimes scientists HARK but that’s a different discussion …)
In this case, it sounds like there truly was some cheating going on of the worst kind (data fabrication).
If some area is getting too many "statistically significant" flukes, they must choose a lower threshold. For example in particle physics they use 5 sigmas that is an insane low threshold, because otherwise they would have to announce a fake particle every month and raise a retraction a few month later.
> even though the coefficient direction/magnitude &c... agreed with the earlier study
This is a problem due to report and publication bias. Last month there was a discussion abut the relation of lead and crime https://news.ycombinator.com/item?id=28016921 . Everybody knows that lead is bad. In that meta analysis, most studies show a very low effect, and only a few have a strong effect. The problem is that a possible explanations is that flukes that show a strong bad effect are published, but flukes that show that plumber is good for people are silently discarded.
Another field with a lot of report and publication bias are drugs against covid-19. For example look at the graph near the bottom of https://news.ycombinator.com/item?id=27852130 How many of these studies are not even statically significant?
It's harmful. This kind of stuff should be handled as quietly as possible, and definitely not in public.
Science is already under attack, we need not undermine it from the inside too.
Creating an actual conspiracy to mislead the public is not a good way to fight this perception.
Acknowledging falsified results is not an attack on science, it is how science works. It shows that science is not a mysterious cult of knowledge conspiring to hide information and deceive others, but rather the collective effort of imperfect humans who want to understand the universe and make things better for us all.
It's bizarre how today, in USA, for some it's all about "the other camp can't have a win, at any cost", and not what is right. Fake science hurts science the most, and covering it up IS WHAT is going to give ammo to whoever you do not like, calling a fraud a fraud doesn't do that.
(compare to medical research, which has embraced prereplication because the FDA is actually serious about safety and efficacy and had the good sense to require it. but the FDA is anchored in the real world - if they get it wrong people may die and it’s on their hands. If psychologists get a result wrong, they may generate a bombastic paper for themselves, do a ted talk, go speak on NPR, then write a popsci book, and generally boost their career. So the incentive is for the FDA to be right and for the psychologist to be wrong, so it’s not surprising that that’s what we see.)