The foot soldiers behind psychology’s replication crisis

There was an interesting study recently that showed prediction markets actually did a better job of determining 'fake' papers than journals in social sciences. [1][2] A group of researchers took some 21 papers published in Nature and Science from 2010 to 2015. 13 were able to be replicated. The prediction market accurately determined all 13 predictable papers, and 5 of the 'fake' papers. They gave the remaining 3 about 50/50 odds. Though one thing to be said there is that even in studies that they managed to replicate, the effect size was found to be, on average, 50% of the stated effect size.

I think there's long been a perception that social sciences are heavily influenced by people who take whatever their biases are, create some experiment specifically designed to show them, and then play with the numbers or experiment's parameters until they manage to do so. This goes all the way back to (and certainly before) Zimbardo's Stanford Prison Experiment. There was nothing organic there. Participants, both prisoner and guard, were heavily coached on how to act and, in their own words, saw the experiment more as an acting role than emergent normal behavior. It seems to be that this perception is accurate.

In a society where people are increasingly radicalizing on social views, we ought expect social sciences to become even more dysfunctional in the years to come. This sort of stuff is, in turn, casting a very dangerous cloud over the rest of science since people tend to extrapolate these actions and behaviors in the social sciences, to science as a whole. In my opinion we need to start creating a strong distinction between science driven by science that yields falsifiability, predictability, and is driven exclusively by direct experimental result -- as compared to not-quite-science that is based on models, abstract experimentation, is not falsifiable, and does not provide meaningful predictions. What I mean by meaningful is that the whole point of predictability is not to have something to encourage political action on as is often the case in social science, but to use as a litmus test for the accuracy of a hypothesis. If it's true then that provides strength to the hypothesis, if it's false then the hypothesis is false. Without falsifiability, predictions are worthless.

[1] - https://www.bloomberg.com/view/articles/2018-08-30/predictio...

[2] - https://www.nature.com/articles/s41562-018-0399-z

Those reviewers sound like internet commenters.

Interesting!

Sadly medicine has a replication crisis. I don't think chemistry or physics does.

That's a criticism of a bunch of heuristics we've had for selecting promising theories. No popular theory has been proven wrong. It's just that we've been stuck for a really long time. And we need some kind of heuristics, otherwise we're just going to overfit to our data.

Yes, that is covered in my wikipedia link and a comment below.

There is a replication crisis in many fields. Not every field.

I've yet to see anything from the current boom be anything more than empirical curve fitting in fancy clothes. I've lived through enough AI winters to not be surprised. Still disappointed.

Pardon my aesthetic judgment, but this is a beautiful comment.

I think computer science is still held hostage to academia if you are interested in doing more creative work. There is a difference between "practical research" where you are developing new ways to solve problems or apply technology and "academic research" where you are developing computer science theory. However, if you want to do practical research, then opportunities are limited if you don't have a PhD. Without one it seems like the primary options are CRUD apps.

It doesn't seem like this is the case in other engineering disciplines or even outside of engineering.

who is us? academic computer scientists?

Premise: > In North America (perhaps elsewhere) you are required to have at least a Master's degree to practise Psychology and you should have a doctorate if you want any mobility with your practise.

Conclusion: > This leads people who have no interest in academia having to find a way to convince people they've discovered something new and novel so that they can go apply what has already been discovered.

I'm not rejecting the premise, I'm saying the conclusion is not supported by this article. None of the figures mentioned in the article (Daryl Bem, John Bargh, Susan Fiske, Brian Wansink, Amy Cuddy, Simine Vazire, etc) are clinical psychologists. None of the research described in the article is clinical psychology, or even appears to have been performed for clinical psychology.

Maybe clinical psychology has a replication crisis, I don't know, but there is no evidence here for the idea that clinical psychology degree candidates are causing the replication crisis in social psychology.

How often are direct replications done in other sciences?

> Freud like all early researchers had no idea what was going on.

I would like to point out that the more you read about this issue, the harder this is to believe.

I hadn't mentioned this, but obviously Freud had a clear financial incentive to enable and hide the sexual abuse that the people who paid for his services committed. And of course, there's the possibility (as some of his patients claim) that he himself also committed those acts, so he may have been protecting himself. (and I don't mean the weird and obviously sexual treatments used, I mean he may have actually raped his patients)

That's one.

Second is that even now there are very atrocious signs about psychiatry, and the other fields of psychiatry, like child services both academic and practice, as both have seen systematic, often legally supported abuse. Just read that link. Some of it, like the abuse in the Soviet Union is in the past.

Most of it is not.

The very basis of child services, the "show me on this anatomically correct doll ..." study was shown to be a fraud. Kids simply point out what they find interesting, and the outcome is random. No correlation to abuse at all. In the study, researchers simply lied about what the kids pointed out. Oops.

How many kids were locked away by child services because they pointed at the wrong part ? There can be no doubt that we're talking tens of thousands minimum.

And of course, https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=2335... . There are no words. The government is unwilling to stop it's use even when fraud was shown. And let's just not mention the question of retracting and fixing of decisions made in the past.

Very similar issues exist for mental services.

On the practice side there is open abuse as well. The main child services decision maker in Norway "used child porn" for 20 years (of course this person has also assigned kids to himself, because of course he did ... note that these children have not been taken away from him after he was found out, nor has he been jailed for what he did. Add to that that the organization he was part of is regularly accused of endemic racism, excessive violence, kidnapping, torture, and worse, causing the deaths of children)

That was 1 year ago. I mean I guess he was doing it for the past 20 years (minimum). But this person got caught 1 year ago. And the system, of course, supported him, after he got caught. How can anyone, at this point, believe that the organization of child services has, in the most generous interpretation, not been infiltrated and is being used to enable and legalize paedophilia, rather than prevent it ? Worst interpretation, of course, is that there never was any other goal. (in Norway, but equally horrifying scandals have occured in the UK, the Netherlands, Belgium, France, Spain and Italy. And I'm pretty damn sure that's because I haven't checked many other countries, or just don't speak the language)

The only reason is "provides a useful foundation of the discipline" is that there are many people whose livelihood depends on this being true ... even when we know it's not just a lie, it was at least partly a bunch of criminals hiding their crimes ... And today, it's no different at least somewhat serve to help disgusting abusers and torturers to commit and hide their deeds.

Nor was it different in the past, as we now know plenty of scandals. Nor was it different under other systems. Democracies, dictatorships, communist states, monarchies, ... it doesn't matter, it happened in all of them. Christian, Muslim, atheist, militant atheist states ... and let's just quickly mention what Japan got caught doing : chemical weapons research on mental patients ... whose status as mental patients was established under very dubious circumstances. None have received compensation, or an apology, because of course not.

> you can spend generations looking into something and still be wildly wrong.

Unfortunately that is very clearly not what happened. Criminals and immoral politicians decided what sort of theory would be useful and the result of that is still used today as authoritative.

And even then we're disregarding the fact that even today results in psychology are so often rather thinly veiled frauds.

https://en.wikipedia.org/wiki/Diederik_Stapel

Note how "trustworthy" this person was. Before he was found out, here's his biography:

  Stapel obtained an M.A. in psychology and communications from the University of Amsterdam (UvA) in 1991.[3] In 1997 he obtained his Ph.D. cum laude in social psychology from the UvA.[3] He became professor at the University of Groningen in 2000[3] and moved to Tilburg University in 2006, where he founded TiBER, the Tilburg Institute for Behavioral Economics Research.[3] In September 2010, Stapel became dean of the social and behavioral sciences faculty.[3]

  Stapel received the "Career Trajectory Award" from the Society of Experimental Social Psychology in 2009

Unfortunately all of this was the result of about 15 years of constant frauds, a fact complained about by double digit numbers of his students, with zero results.

Needless to say, his book is used by child services and in the Netherlands, and they have not made any retractions, apologies, or changed any of their decisions (which result in incarcerating children) as a result of this.

At what point do we say "this is a total failure" and call these people off ? I guess they're just too useful for corrupt politicians.

Well, there's hypothetically a way to find out:

https://news.ycombinator.com/item?id=16947652

if after introduction of such a system, the number of "hysteria" diagnoses or whatever modern equivalent starts dropping substantially, it would prove Rush right.

Benjamin Rush successfully pursued a libel claim against a man who wrote that his bloodletting killed more patients than it ever helped. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1312212/

> Rush continued to advocate his depletion therapy during the yellow fever epidemics in Philadelphia in 1794 and 1797, although his reputation and practice were already waning. By 1797, William Cobbett, the satiric journalist who frequently targeted Rush, was in full cry. He reviewed the 1793 bills of mortality for Philadelphia and showed that the mortality rates increased significantly following the institution of Rush's remedies. He characterized Rush's work as “… one of those great discoveries which have contributed to the depopulation of the earth.” When Rush referred to calomel as the “Samson of medicine,” Cobbett wrote:

>> Dr. Rush in that emphatical style which is peculiar to himself calls mercury the Samson of medicine. In his hands and those of his partisans it may indeed be justly compared to Samson: for I verily believe they have slain more Americans with it than ever Samson slew of the Philistines. The Israelite slew his thousands, but the Rushites have slain their tens of thousands (28).

> Rush sued Cobbett for libel in 1797. The case dragged on for 2 years, probably due to political maneuvering by Rush's enemies. Cobbett was found guilty and fined $5000 (later reduced to $4250), at the time the largest award ever made in Pennsylvania. The damage had long since been done, however, and Rush's practice had vanished by 1797.

How many different ways are there to form a solar system? I am sure the requirements for experiment and possible combinations of factors are just as daunting.

So, I don't think thats the real problem. I think the main real problem is that in the failing fields they look for differences between groups while in the successful fields they look for "universal" rules.

> I'm of the opinion that the complexity of possible human behavior and phenomena is too great to allow for some experiments to be replicated and controlled.

If these elements of human behavior cannot be confirmed by replicable experiments, what chance do we have of knowing about it. Claims about such behavior are nothing more than stories, for they are not based on evidence.

> There are some scientific facts about humans that you can establish because you can do a replicated, controlled experiment and there are others that you can't

There is no such thing as a 'scientific fact' that cannot be established by a replicated controlled experiment. The stated dichotomy is really important though, and one that it seems psychology has partially failed to make. For it lost its reputation by presenting 'stories' as scientific fact.

That is not to say there is no value in studying human behavior that is beyond science, but we need to realize that we cannot treat the result of this as 'scientifically true'. Instead, it is something like 'intuitively true based on anecdotal experience'.

An interesting point and not untrue, but it's only important if we're trying to get at "human nature" in itself, which to me seems kind of silly as we're social animals by nature. In practice, there's nothing wrong with testing different levels of awareness, and if an experiment is part of popular culture then it's part of us too.

It should be noted that the SPE's methodology was totally fucked in the first place, and before reproducing it we'd need to actually perform it "right" at least once (which can't be done now that ethics* boards are a thing).

Good comeback, sad that it's relevant too. I've seen entirely too many sketchy books and articles attempting to link quantum physics and psychology.

I was in academia dealing with the same stuff. I worked constantly for years to save my project once I figured out what was going on. I got it to the point where at least it wasn't BS, but after that was too burned out to go further. At every step there were social (not scientific) obstacles to me doing a good job.

After completing the degree, I quit rather than produce fake science or spend my time trying to fight against those who are. I have been there, and have no pity for people who choose to produce fake science.

Sure, in some areas of physics they reproduce studies millions or billions of times by figuring out how to get it done cheaper and cheaper until students can do it:

https://hepweb.ucsd.edu/2dl/pasco/Millikans%20Oil%20Drop%20M...