Psychologists Welcome Analysis Casting Doubt on Studies | Hacker News Reader

Psychologists Welcome Analysis Casting Doubt on Studies

nytimes.com

37 pointsevolve2k26 comments

How much of this is the result of outright fraud? I know Hanlon's razor and all, but I find it extremely difficult to believe that virtually everybody in this field is incompetent.

It seems like the majority of these highly intelligent people, despite spending a decade in training, were essentially statistically illiterate, even though statistics are absolutely central to their field. That's not a catastrophe in itself, you can always hit up the "real" statistician two floors down...but it seems like they didn't want to bother with that. Did they overestimate their ability or did they know they were serving up shitty research?

The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)...I'm not sure how to interpret this. Did they get lucky and stumble upon the one area of psychological research that had a "high yield"? Did they implement a superior research methodology that nobody could/wanted to imitate?

I've always felt like Hanlon's razor needs some kind of clause to account for corporatacracy. It's difficult to figure out how to attribute enforced monopolies, which take a great deal of planning and political cunning to establish, to incompetence.

Where I study psychology, I'm noticing that the pace of learning statistics is slower compared to statistics classes in my CS program. The fact that I'm doing multiple programs is a hint in itself. For example, we receive lessons about normality assumptions but do not receive lessons about non-parametric statistics. Furthermore, whenever I gave critique about these kind of things they were always shoved away as if they were a minor inconvenience.

Have a look at this what this Harvard neuroscientist says about failed replication. It's pretty scary.

http://web.archive.org/web/20150115080542/http://wjh.harvard...

Well that makes me think the fraud angle is fairly likely...

It made me think of incompetence, but I guess fraud fits too.

The rejection of scientific method was what scared me.

I don't know about neuroscience in particular, but most professions have a code of ethics that forbids you from working in an area where you're not professionally qualified. This specifically addresses the gray area between incompetence and fraud.

I find this scary because the author seems to attribute negative failure of replication to incompetence in a way that he doesn't treat the original success, and essentially ends up rejecting all failed replications - how many good results are sufficient vs how many bad results?

Is that what you meant?

Yes. It's a scary rejection of the scientific method.

How did that guy become a professor of psychology at Harvard?

(The archived link itself is dead, but he's listed as a PI here: http://www.wjh.harvard.edu/~scanlab/people.html)

If people cannot replicate your methods the fault is yours, period. The claim that it is "too hard" is bullshit. Write better methods. Science is nothing without replication. There are plenty of people writing vague descriptions of their methods to protect their status. These people are shit scientists.

Not to try to derail the conversation, but I have noticed a very poor understanding of statistics and experimentation in all of the "soft" sciences, in particular, the social sciences.

It is not uncommon to see published and "peer reviewed" work that makes sweeping claims off a tiny amount of self-reported data with nothing else to back it up. I have seen plenty of stuff that people on the "hard" sciences side of campuses would be embarrassed to even take responsibility for ... not to say they don't try to slip one by every now and again, but it seems systemic in the "soft" sciences.

This is endemic to all the sciences, not just the social ones. Biology has just as poor (if not worse) an understanding of stats, and even worse pressures in terms of the money at stake (medicine, pharmaceuticals).

Maybe physics, and to some extent, chemistry, are immune, I'm less familiar with them.

The issue is less stats, and more the pressure to publish, file drawer effects, poor career prospects for replication work relative to original research, fishing without multiple comparisons correction, etc. etc. etc. All of which combines to result in a lot of scientists looking the other way because the rules for stats and the rules for tenure aren't in alignment.

I saw all this first-hand when I was a grad student in cognitive neuroscience. Don't for a second believe this is limited to psychology. If anything, psychology is being more honest and grappling with these issues first.

It doesn't require any malicious intent. An alternative possibility is that the things they were researching just weren't likely in the first place. If an effect that only has a 5% chance of being true in the first place, then even if you confirm it with a 95% confidence the chance that it is a fluke could still be as high as 50%.

This can be made worse if you test several different hypothesis, only report successful experiments, and don't replicate them.

When I look at the people I know personally who study psychology, sociology and related fields and see what they learn, I don't have any problems with believing that virtually everybody in the field is incompetent.

It starts with the little things. I remember bringing up on HN the bullshit surveys most students do for their Master's thesis that are statistically worthless for the reasons said students don't understand. I've been told that hey, it's MSc, not a PhD, it doesn't have to be research-quality, etc. But hell, if people are willing to rely on bullshit methodologies for their Master's, when exactly do you think they'll learn the right way of doing things? Does the statistics god dispense additional levels of enlightenment when someone crosses the PhD threshold? I don't think so.

And honestly, it's not that much better in the hard sciences as well, on the undergraduate level. Maybe they won't let you get away with bullshit stats, but they'll happily let you get away with fabricating results and tweaking data to fit the model. Studying today is about only one thing - do everything you can to pass the exam. Research ethics? Never heard of it.

I know people in the life sciences who admit to frustration over the pressure to publish and the abuse of statistics.

But I don't know if statistics is the entire problem. It seems to me that a feature of good work in the "hard" sciences is that a problem will be approached from a bunch of different angles, so that the result doesn't really depend solely on statistics.

Synthetic chemists have to perform multiple orthogonal tests in order to announce a new compound, such as IR, NMR, X-Ray crystallography, and elemental analysis, on top of describing a mechanism for the compound's formation and performing reactions with the compound if possible.

Traditionally, to prove discovery of a new element, it wasn't good enough to see a blip on a mass spec. You had to do chemistry with it.

In my view a problem with data driven science is that you can do statistics without even knowing what you're actually studying. But if you don't know what you're studying, then you can't attack a problem from multiple angles because you don't even know if your tests are all measuring the same thing. You also have no tools to ask the hard questions about whether the results really make sense.

I don't think it's just this field.

Statistics are routinely abused in nearly every field of science, alas.

The OP mentions findings about how forcing pre-registration on clicnicaltrials.gov seems to change results. Those are, of course, medical research studies. A discussion of those findings regarding pre-registration was on HN 10 days ago. https://news.ycombinator.com/item?id=10080680

I do not think it is conscious 'fraud' exactly. Sloppy statistics have just become routinized in standard scientific research in many fields. And yes, that sloppiness almost always errs in the direction of making it easier to show positive results, of course. But I don't think the researchers realize they are doing something wrong exactly, it's just totally routinized.

> The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)

This doesn't surprise me at all. Kahneman wrote a book, "Thinking, Fast and Slow", in which he spends a long time talking about how most psychologists and social scientists do not understand the basics of statistics and therefore end up with flawed conclusions.

"One piece of evidence that preregistration can act as a strong corrective comes from clinicaltrials.gov, a registry of publicly and privately funded clinical studies involving human subjects. Before 2000, when the site was established, 57 percent of large clinical trials funded by the National Heart, Lung and Blood Institute showed a significant benefit of drugs or other intervention, according to a recent analysis published in the journal PLOS One. After the registry was put in place, only 2 percent of such trials found a clear benefit."

I think it's healthy to suspect that the entire pharmacopoeia-industrial-complex has a similar rate of veracity.

I'm absolutely sure about that. As a student I was working (as a technician) on many medical science conferences, so I had an opportunity to see the "bleeding edge" research presentations. I was shocked by the level of confidence demonstrated by "scientists" who promoted results of their N=5 experiments (without a comparison group) as "convincing". If all therapies are based on such research, we have to be happy that our bodies are impressively self-repairing machines.

I'd recommend Irving Kirsch's __Emperor's new drugs: Exploring the antidepressant myth__ as an accessible introduction to this topic.

The book goes beyond statistics and draws attention to "clinical significance" that is crucial to keep in mind when evaluating drugs. Statistical significance helps to inform whether a metric has changed; clinical significance tells whether it actually mattered from a patient's perspective.

Dr. Kirsch's (a psychologist) work has been valuable, with some justified criticisms of antidepressant therapy and their trials. But his work did not show that antidepressants are clinically ineffective.

Dr. Kirsch's analysis showed that for mild depression treated with antidepressants, there is in fact an improvement in symptoms of depression based on rating scales like the Hamilton Depression Scale - but for mild depression, this improvement in the numbers doesn't necessarily translate into what you correctly identify as clinical significance. However, Dr. Kirsch's own "cutoff" for whether or not such a numerical increase constitutes "clinical significance" or not is itself just an arbitrary number that itself has been criticized.

Also, Kirsch's own metaanalyses (and other metaanalyses) show clearly that for moderate and severe depression (and other conditions), antidepressants are indeed clinically effective (in other words, yes, the benefits actually "mattered" from a patient's perspective), and, importantly, they are highly effective at preventing relapse.

They work, and work well.

In addition, there are a variety of reasons why many short-term clinical trials of antidepressants (which have made up the bulk of clinical trials of SSRIs, for example) are entirely different from the way physicians prescribe antidepressants (and other medications) clinically, and may underestimate their benefit even for cases of mild depression. The beneficial effects of antidepressants increase with longer exposure (i.e. exceeding the length of a six week clinical drug study), probably because of downstream neurotrophic factor(s) increases. Many trials of antidepressants limited dose adjustment or dose adjustment rate. Many clinical trials have excluded patients with severe symptoms or dual diagnoses because inclusion introduces a safety issue or may introduce too many variables. Also, physicians will switch patients to another antidepressant if there is no sign of early efficacy, and while one antidepressant may work very well (probably as a result of genetics), another will not work at all. Most studies focusing on one drug are not going to switch to another medication in the same class (for example, a serotonin reuptake inhibitor) during the trial, even though that's routinely done in clinical practice with good result.

Antidepressants work. Psychotherapy can also be helpful as a sole treatment for mild depression (although please note that psychotherapy has its own set of risks, costs, and drawbacks), and patients often get additional benefit by engaging in psychotherapy along with medication treatment. However, a psychologist or counselor treating a moderately to severely depressed person who fails to refer that patient for evaluation for treatment with an antidepressant would be risking a malpractice suit.

At the risk of sounding really stupid, What is the crowning achievement of psychology ? What gives this field the legitimacy that it enjoys ?

Well, we we now understand a lot more about how people think than before. We're finally past that id-ego-superego and black box theory bullshit (although some people still ascribe to freudian ideas). There is real treatment for people with mental illnesses other than hospitalization and drugs (e.g. people with personality disorders, dissociation disorders, etc. can now receive structured talk therapies)

The crowning achievement would probably be the ability for people who are mentally ill to receive help and understanding from trained professionals rather than be considered possessed, needing drugs, shoved into insane asylums, or eugenics'd.

I would say that one of the most important "meta" ideas, that comes up in many places, is that your conscious mind and rationality are largely a veneer/illusion over an "animal" brain.

Emotion comes first, then thought. This has consequences in the real world, including economic ones. And it was something that people got wrong -- e.g. Enlightenment thinkers overstated how different and privileged humans are in this respect.

Take the example of optical illusions. Even though I can tell how the illusion works, and you rationally know that two lines are the same length, and you can literally measure it yourself with a ruler -- your mind will still see them as different [1].

This is a benign example, but there are lots of other examples where "knowing" that you are wrong doesn't prevent you from acting as if you held the wrong beliefs.

If you are an investor, you should be familiar with the concept of "loss aversion", which is easily verified with an experiment and replicated. Daniel Kahneman won a (deserved) Nobel Prize for many interesting phenonema with real economic consequences.

Intelligent people are always questioning their psychology: http://www.paulgraham.com/swan.html

The first rule I knew intellectually, but didn't really grasp till it happened to us

I don't know whether these expectations are innate or learned, but whatever the cause, we are just not prepared for the 1000x variation in outcomes that one finds in startup investing.

I assert that certain ideas from psychology have permeated into the public consciousness -- and 200 or 300 years ago, you would not hear non-psychologists talking in this fashion.

You can do very repeatable psychological experiments on how people perceive quantities of numbers and so forth. For evolutionary reasons, there are lots of quick "short circuits" in your brain that perform resource allocation and risk assessment, but they are not adapted for the modern world.

[1] https://en.wikipedia.org/wiki/M%C3%BCller-Lyer_illusion

Reply on news.ycombinator.com