Comment by Certhas | Hacker News Reader

Comment by Certhas | Hacker News Reader

Parent Full thread

Or it wasn't, e.g.:

http://neuroneurotic.net/2016/03/03/the-non-replicability-of...

From where I’m standing, social and other forms of traditional psychology can’t say the same. Small contextual or methodological differences can quite likely skew the results because the mind is a damn complex thing. For that reason alone, we should expect psychology to have low replicability and the effect sizes should be pretty small (i.e. smaller than what is common in the literature) because they will always be diluted by a multitude of independent factors. Perhaps more than any other field, psychology can benefit from preregistering experimental protocols to delineate the exploratory garden-path from hypothesis-driven confirmatory results.

I agree that a direct replication of a contextually dependent effect in a different country and at a different time makes little sense but that is no excuse. If you just say that the effects are so context-specific it is difficult to replicate them, you are bound to end up chasing lots of phantoms. And that isn’t science – not even a “soft” one.

The article explicitly says that the 39% reproducibility isn't that far off what you would probably expect anyway.

The article then goes on to add the the meta-study was awful in it's study selection, giving the example: OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam.

The article then goes on to say that reproducibility is still a problem in science, but that the meta-study was simply terrible in it's own methodology. Low reproducability is less of an issue given the context of the problem space, the article says; the meta-study was wrong, both methodologically and in conclusion ('low reproducability' != 'unethical' or 'crisis').

I recommend reading the fine article.

>OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam.

If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine. If they did conclude something on "Stanford University students", then it is a bad reproduction. You see, there's a very good justification there, that you can interpret as subtyping, or the liskov substitution principle.

>'low reproducability' != 'unethical'

This is akin to saying "ok there's this theorem, but I won't give proof, just a bogus heuristic argument", or even worse, saying "that person is guilty but I won't justify why". Both these behaviors are unacceptable. Why would "social sciences" be above that?

> or 'crisis'

A field is filled of politically oriented baseless claims, but this is not a crisis, just another day in Oceania.

That said, if the meta-study was sloppy, then it still justifies that research standards should have been upped a while ago; the "terrible"" meta-study was accepted precisely because terrible papers are accepted in the first place.

> If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine.

No. Just fucking 'no'. It was a study on affirmative action policies, repeated in a distant country with different public opinions on welfare and social fairness. If you think that that's fine, then you really have no claim to be declaring what psychology should and shouldn't be.

> filled of politically oriented baseless claims

Fuck me, but it's funny when detractors claim that psychology is full of unscientific behaviour, while they're way overgeneralising the entire field on a subsection of it.

So you say 'no' but you don't refute the argument? Can you explain why following correct substitution principles does not work?

You could also try to answer to the arguments of harry8, who gave you polite and well-written points, that you consciously ignored and resorted to name-calling ("you're heavily clueless about what psychology is") when you have absolutely no element to back your claim.

>Fuck me, but it's funny when detractors claim that psychology is full of unscientific behaviour, while they're way overgeneralising the entire field on a subsection of it.

What I mean by filled is that this subsection (which exists in any field, even mathematics) is an important fraction of this particular field. And before you bring this up: I read a lot of important papers in the field, and they were all below my expectations, which were admittedly not very high. I came to think that the "journal of irreproducible results" is one the most serious publications in soft sciences.

>Apparently arguments become stronger when you add "fuck" qualifiers

Fuck me!

His point was about whether the term "students" is qualified.

If that's the reason for including the study, then it was a bad study to choose to replicate, as it would be clearly not replicable for students of other cultures - in which case, the meta-study researchers are stacking the deck; they would be guilty of the very thing they're decrying.

It's interesting to compare that link talking about how it's difficult to replicate psychology (and other) experiments, with this link that says it doesn't matter.

https://web.archive.org/web/*/http://wjh.harvard.edu/~jmitch...

(I agree much more with your link).

There are many things wrong with the analyses performed by Gilbert et al. Sanjay Srivastava does a decent job at pointing them out:

https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-cr...

Interestingly, variations on "the exploratory garden-path" are widely hailed by qualitative social science researchers as sound methodology.

Reply on news.ycombinator.com