Classic and contemporary psychology experiments replicated | Hacker News Reader

Classic and contemporary psychology experiments replicated

cos.io

77 pointsmathgenius34 comments

Short summary:

- The 28 findings was repeated in more than 60 laboratories all over the world resulting in a median sample size of 7,157

- Overall, 14 of the 28 findings (50%) replicated successfully

- The replication team obtained the original materials to ensure faithful replications of the original findings

- All 28 studies underwent formal peer review at the journal prior to conducting the studies

- All of the studies were preregistered at OSF

- Researchers can predict whether findings will replicate or not in surveys and prediction markets with pretty good accuracy. (prediction market)

Full paper (100 pages): https://psyarxiv.com/9654g/

"It was surprising that even with these efforts we were only able to obtain support for the original findings for half of the studies. These results do not definitively mean that the original findings were wrong, but they do suggest that they are not as robust as might have been assumed." - Michelangelo Vianello, one of the project leads.

Yes, yes they do. It's ok to say the original findings were wrong. It's ok to re-assert the null hypothesis. The unwillingness to do so undermines the process and is disconcerting to say the least.

Failure to replicate doesn't prove the original findings were wrong. Other things that can cause replication failure are:

- Poor explanation of experimental procedure, so the replication doesn't match the original experiment.

- Awareness of the effect may invalidate it. For example, it'd be hard to replicate the Stanford Prison Experiment because every psychology undergrad has heard the story.

- Change in social conditions or prevailing attitudes. Consumer behavior (the subject of some of these studies) has changed significantly over the decades. Homophobia (another study) varies dramatically over time and place.

We should continue this argument over coffee, hope you're doing well!

Please record the audio, i'm curious who convinces who.

(or just continue here on HN of course)

Not exactly: it is ok to say that there were no original findings. This is a tiny semantic detail but important: if the author does not support his findings, it is OK to re-assert the null hypothesis but not to say that "the findings were wrong".

What I mean is: "no experiment" does not mean the same as "wrong experiment".

Well, let's agree that 'wrong' doesn't mean 'bad.' I use the term in the Feynman sense, if your guess doesn't agree with experiment, it's wrong. The assertion that they found a statistically significant effect was wrong (or to Trevor's point in a sibling comment, is no longer true.)

They are saying that they made have mistakes now, it's rather defensive. Isn't it?

Fun game: read through the summary of each experiment (psyarxiv.com/9654g) and try to guess whether it replicated or not before they tell you. I got 20/28.

Some of them were obvious (#3 - no, using a bad font does not make people answer questions more carefully; #22 - yes, people attribute more responsibility to adults than to babies). But there were some big surprises for me:

#6 - Participants read a hypothetical water conservation dilemma, then rate how much they trust the people in the scenario to conserve water. Sometimes the people are called "consumers" and sometimes "individuals". Does it make a difference? Apparently yes. When they're called "consumers", you expect them to conserve significantly less water than when they're "individuals".

#24 - if you ask "How satisfied are you with your relationship?" and then "How satisfied are you with your life-as-a-whole?", are the results more highly correlated than if you ask those questions in the opposite order? I thought for sure they would be - relationship happiness is one of several contributing factors to your overall happiness, so if it's fresh on your mind, you'll weigh it more heavily when you answer the life-as-a-whole question. Apparently not.

I'm about halfway through doing just that and I think a pattern has emerged: does there exists a plausible, concrete (you could put it into words) mechanism for cause and effect? That is, can a person tell you why the finding exists. For instance, a semantic association between capitalism and greed does not provide a plausible mechanism for why mentioning capitalism should make one greedy. The gay one failed that test, but otherwise that seems to be what's going on. A lot of these papers aren't testing a scientific theory. They're testing vague, half-baked ideas and associations.

List of studies, and the outcome:

1 Direction & SES (Huang et al., 2014) p<.001

2 Structure and goal-pursuit (Kay et al., 2014) p=0.97

3 Incidental disfluency (Alter et al., 2007) p=0.67

4 Moral Foundations (Graham et al., 2009) p<.001

5 Affect and Risk (Rottenstreich & Hsee, 2001) p=0.77

6 Priming consumerism (Bauer et al., 2012) p=0.15

7 Correspondence Bias (Miyamoto & Kitayama, 2002) p<.001

8 Disgust & Homophobia (Inbar et al., 2009) p=0.56

9 Incidental Anchors (Critcher & Gilovich, 2008) p=0.25

10 Social Value Orientation (Van Lange et al., 1997) p<.001

11 Trolley Dilemma 1 (Hauser et al., 2007) p<.001

12 SMS & Well-Being (Anderson et al., 2012) p=0.58

13 False Consensus 1 (Ross et al., 1977) p=0.23

14 False Consensus 2 (Ross et al., 1977) p<.001

15 Position & Power (Giessner & Schubert, 2007) p=0.31

16 Framing (Tversky & Kahneman, 1981) p=0.43

17 Trolley Dilemma 2 (Hauser et al., 2007) p=0.42

18 Tempting Fate (Risen & Gilovich, 2008) p=0.01

19 Actions are Choices (Savani et al. 2010) p<.001

20 Intuitive Reasoning (Norenzayan et al. 2002) p<.001

21 Less is Better (Hsee, 1998) p<.001

22 Moral Typecasting (Gray & Wegner, 2009) p<.001

23 Moral Cleansing (Zhong & Liljenquist, 2006) p=0.08

24 Assimilation and Contrast (Schwarz et al., 1991) p=0.39

25 Choosing or Rejecting (Shafir, 1993) p=0.10

26 Priming warmth (Zaval et al., 2014) p=0.01

27 Intentional Side Effects (Knobe, 2003) p<.001

28 Direction and Similarity (Tversky & Gati, 1978) p=0.99

Let me know if I messed anything up and I'll fix it, it was a quick copy-paste.

which journals would be nice. quick way of getting idea of how important these papers actually were.

None of the classics.

I'm not much of a psychologist, but I can see at least one paper by Khaneman, who won the economics Nobel for that sort of stuff. So not sure if you're being sarcastic or not?

20 of the 28 are from this decade which hardly makes them classics. Nothing from the 50s, for example. This list isn't The List.

To be fair, the title says classic and contemporary. Probably would have been better to just say "High Profile" or something instead.

Nobody wants to touch some of the classics, at least in the USA. Like this one:

https://en.wikipedia.org/wiki/Pit_of_despair

Scientifically, it might be of value to replicate the experiments, but I really can't imagine anybody attempting them.

it's almost as if they are avoiding all the experiments that have been replicated so often the result barely bears mention in the papers using them in favor of random crap from the field.

Looking through the paper, even the half that replicated generally seem to have done so with greatly diminished magnitude.

Poor choice of headline for the original article given 'replicated' indicates success. This was an intensive attempt to replicate but 14 of the 28 did not replicate.

That is a 50 percent failure rate inspite of using over 186 researchers in 60 labs worldwide. The widely reported 2012 study could not replicate nearly 100 of 100 studies. And in this case if this kind of comprehensive effort can't replicate half the studies it raises serious questions about the original studies and adds more fuel to the reproducibility crisis.

Given studies are routinely used to justify and implement policies [1], and taken seriously as points of authority it's not just academic.

There needs to be much larger discussion on the use of studies to direct policy that impact real lives, and the level of oversight, additional replication and other credibility building measures that need to be considered.

[1] https://www.sciencedirect.com/science/article/pii/S001632871...

The thing with psychology experiments is that the fundamental thing being tested is not immutable.

This makes it different than many other science experiments where concrete parts of reality are tested.

Psychology really only tests the minds of humans in a global society at a point in time. If future minds change enough the results may no longer be valid.

Your statement in itself is a hypothesis to be proved. There are psychology experiments designed specifically to ascertain what parts of the mind are hard-wired and which are depended on nurture/culture.

I am not a Psychologist but I can give one example: Piaget hypothesized that the mind developed by first accumulating low-level sensory/motor data, and then progressively building higher levels of abstraction on that foundation. If that is true, then that process itself is not related to society or culture and should be observable so long as the biology underpinning cognitive development doesn't radically change.

The mind is mutable.

Psychology is different than other "hard" sciences.

I only say "results _may_ no longer be valid" and I'd say this study proves that.

If you go deep enough into the mind to the point where social environment no longer influences the behavior you may be getting closer to biology than psychology but point taken.

>This makes it different than many other science experiments where concrete parts of reality are tested. //

Maybe, or maybe immutability is axiomatic in those other areas.

Here are some of the details of some of the unreplicated studies (not scientifically specific - but you can always read the paper)

Replication Failures

- Disgust sensitivity predicts homophobia (no it doesn’t)

- Assimilation and contrast - the order of asking specific vs. general questions about the same domain matters (no it doesn’t)

- Priming Consumerism - being told you’re acting as a consumer reduces your trust in others (no it doesn’t)

- Moral Cleansing - people exposed to unethical things have an urge to cleanse themselves physically (no they don't)

- Priming Warmth - on warm days people have higher belief in global warming (no they don’t)

- Structure & Goal Pursuit - being exposed to structure improves goal pursuit (no it doesn't)

- Incidental Disfluency - making tasks incidentally difficult (hard to read fonts) - improves analytic performance (no it doesn’t)

I'd love for someone to go through and do this with education research. It's an absolute joke, and I'd argue it's worse than psychology research. And it doesn't help teachers are making students refuse to use old literature (or maybe I'm just pissed I can't quote older studies in my 'literature review' for my 'research paper'...)

I'm just going to point out that the ridiculous idea that using a hard to read font improves retention of information was disproven.

Why do you class that as "ridiculous"? A mechanism by which that might be true leaps readily to mind.

Because the science behind building better instruction already exists and does not rely on parlor tricks to make poor instruction more memorable.

Is there a list of the experiments and the effect sizes and how they differed from the original studies?

Yes, in the paper, you can to follow the links.

I read psych papers. Regularly. I don't know any of these aside from Tversky's.

Psychology is a big field. I just find it hard to accept that these papers are representiative of much of anything.

TLDR; only 50% succeeded replication.

You don't have to feel so bad about not reading that Malcolm Gladwell book, now.

Reply on news.ycombinator.com