In a showdown of psychotherapists vs. ChatGPT, the latter wins, new study finds

Here’s the study itself. It is pretty cleverly designed, even though it can’t be compared to a therapeutic relationship—at all.

People basically signed up to get couples help — and received a message. Then, they had to guess whether the response they got was from ChatGPT or a therapist; people were close to chance!

Then, still without knowing, they had to rate the quality of the response.

https://journals.plos.org/mentalhealth/article?id=10.1371/jo...

It's the usual low-quality, clickbaity "scientific" reporting that isn't that much present in the research article. Unless you are very careful about saying that it's "a" showdown and does not imply that that ChatGPT is better than therapists. (There so many others showdown where ChatGPT is better than psychotherapists, such as speed-reading, connecting to the internet, getting care from OpenAI -- but these aren't really relevant).

To be able to show that ChatGPT can do better, a study would need to consider much longer term interactions, where memory of the past conversations is important, to keep track of what worked, why, and how to adapt the strategy in response. Furthermore, everybody knows that "you're valid sweety" is going to be a much preferred reply than a harsh truth. So preference cannot be taken as a proxy for efficacy or anything really.

The paper, "When ELIZA meets therapists: A Turing test for the heart and mind", does not really make such claims, but says that "This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes.", since AI-generated responses were generally better received. By improving short-term interactions, they could get better results on the long term.

I am as skeptical of psychotherapists as can be (also skeptical of AI and about skepticism), and I'm willing to believe that AI can do better -- if only because this is the nature of statistical aggregates. But this isn't enough to show anything, unless you just enjoy your confirmation bias.

This story is written to appeal to people who have no concept what a "therapeutic relationship" is.

> “participants in most cases preferred ChatGPT’s take on the matter at hand.”

That doesn’t come as a surprise to me. I’ve tried several modalities and intensities of therapy from CBT and DBT to Psychoanalysis, and I find ChatGPT more beneficial. A crucial benefit of ChatGPT is its 24/7 availability. By reducing the barrier to availability, it becomes much easier to get help as unhealthy beliefs and behaviors emerge. Perhaps, LLMs could augment clinician understanding of the environmental factors that trigger patient symptoms.

Even if it turns out to be worse, having an alternative can be good. Imagine a bunch of therapists in [insert country you find evil], they obviously depend on salaries and for example could divert critical but healthy sentiments toward e g the local military. Strange example but just the cultural variation and independence I mean can be worthwhile.

But then there’s the logs and the training and telemetry maybe a journalist for example would have to run locally if they had such a demand without better alternatives

It's ELIZA[1] before; now, it's ChatGPT.

From technology point-of-view, Humanity have progressed a lot; however, psychologically speaking, we're still almost the same as we're then (in 1960s).

--

[1] - https://en.wikipedia.org/wiki/ELIZA

It doesn’t matter if people prefer the AI therapist. If you ask whether people prefer a doctor who gives them any drug they request without question, the answer will be yes, too. This is not in the public interest, though.

Of course, questions of efficacy haven’t been answered, but even if it seemed efficacious, it is still irreducibly irresponsible. It is literally irresponsible because AI cannot be held responsible for its acts, and neither will its creators (although they should).

AI is prone to wild breakdowns, in a way that humans are not, and only very large and long studies could assess that empirically. And even if you did such a study, it would apply only to that model of LLM using that exact prompt set, and those particular people with those problems.

Neat. No therapeutic outcomes, so in this highly constrained context... sure.

People were attached to Eliza, and ChatGPT will probably be better than a bad therapist, but if you've got the resources to hire a real professional... do that.

I found incredible how is not included in the discussion the confidentiality. The kind of information a patient share with a professional is sensitive. I don't consider something to be stored or manipulated on any way by a wrapper AI service or even an AI provider that potentially to be exploited for profit

Yeah I'd take this with a HUGE grain of salt:

    Next, when examining whether responses written by ChatGPT and therapists were rated higher, lower, or equal in line with the common factors of therapy, we found a large and reliable difference (d = 1.63, 95% CI [1.49, 1.78]) favoring ChatGPT. Given that common factors undergird and act as a mechanism for much of the evidence-based treatment literature [27, 28], and the direction and size of the effects, consulting with ChatGPT may be a way to improve declining or stagnating effect sizes within the evidence-based clinical psychological sciences [34, 35].

    The second aim is likely to be critiqued for several reasons. First, these are responses to therapy-like vignettes and may not generalize to actual therapy. These responses represent a hypothetical “snapshot” of therapy. Second, vignettes were based on couple therapy scenarios and the results might not generalize to individual therapy. Third, the outcome measured was brief and might not fit neatly into varying definitions of what is (or is not) therapeutic. Other criticisms might be with the clinician sampling plan, how responses were selected for comparison, or the therapist group. Future work should include clinicians from different backgrounds, modalities, and theoretical orientations to solidify these findings. Including more diverse therapists may produce responses perceived as more therapeutic than those written by a large language model.

They also win in price, convenience, unlimited patience and availability. Although the human touch physically is almost impossible to replace

I've found ChatGPT and Claude to be immensely helpful for my mental health. I made breakthroughs with them that I haven't in years of therapy.

...at the same time, it's severely limited in important ways. And the lack of real human connection is a problem.

I think well tuned AI in conjunction with a trained and empathetic therapist has a lot of promise. I don't think it's an either / or thing, they can work together.

I'm actually overall bearish on AI in many ways, but this is one that I'm pretty confident in.

Obviously this is no replacement for a high quality therapist. Good luck finding one.

There are a lot of low quality therapists that cause harm to patients through incompetence. AI has a fit in the market somewhere.