It's a bit like talking about the quality of pastoral care you get at Church. You can get a wide spectrum of results.
Worth pointing out such systems have survived a long long time since access to it is free irrespective of the quality.
Worth pointing out such systems have survived a long long time since access to it is free irrespective of the quality.
No, no it isn't.
Whatever you think about the role of pastor (or any other therapy-related profession), they are humans which possess intrinsic aptitudes a statistical text (token) generator simply does not have.
And an LLM may be trained on malevolent data of which a human is unaware.
> The question is not if they are equals, the question is if their differences matter to the endeavour of therapy.
I did not pose the question of equality and apologize if the following was ambiguous in any way:
... they are humans which possess intrinsic aptitudes
a statistical text (token) generator simply does not have.
I think you're wrong, but that isn't really my point. A well-trained LLM that lacks any malevolent data, may well be better than a human psychopath who happens to have therapy credentials. And it may also be better than nothing at all for someone who is unable to reach a human therapist for one reason or another.
For today, I'll agree with you, that the best human therapists that exist today, are better than the best silicon therapists that exist today. But I don't think that situation will persist any longer than such differences persisted in chess playing capabilities. Where for years I heard many people making the same mistake you're making, of saying that silicon could never demonstrate the flair and creativity of human chess players; that turned out to be false. It's simply human hubris to believe we possess capabilities that are impossible to duplicate in silicon.
The scale needed to produce an LLM that is fluent enough to be convincing precludes fine-grained filtering of input data. The usual methods of controlling an LLM essentially involve a broad-brush "don't say stuff like that" (RLHF) that inherently misses a lot of subtlties.
And even more, defining malevolent data is extremely difficult. Therapists often go along with things a patient say because otherwise they break rapport. But therapists have to balk once the patient dives into destructive delusions. But data of a therapy can't be easily labeled with "here's where you have to stop", just to name one problem.
A simple good search reveals ... this very thread as a primary source on the topic of "malevolent data" (ha, ha). But it should be noted that all other sources mentioning the phrase define it as data intentionally modified to produce a bad effect. It seems clear the problems of badly behaved LLMs don't come from this. Sycophancy, notably, doesn't just appear out of "sycophantic data" cleverly inserted by the association of allied sycophants.
In the context of this conversation, it was a response to someone talking about malevolent human therapists, and worried about AIs being trained to do the same things. So that means it's text where one of the participants is acting malevolently in those same ways.
For me, hearing this fantastical talk of "malevolent data" is like hearing people who know little about chemistry or engines saying "internal combustion cars are fine long as we don't run them on 'carbon-filled-fuel'". Otherwise, see my comment above.
The thing they're talking about is hard but it's not impossible.
You can do it pretty practically. Figuring out a supply is probably worse than the conversion itself.
> My point is that mobilizing terminology gives people with no knowledge of details the illusion they can speak reasonably about the topic.
"mobilizing terminology"? They just stuck two words together so they wouldn't have to say "training data that has the same features as a conversation with a malevolent therapist" or some similar phrase over and over. There's no expertise to be had, and there's no pretense of expertise either.
And the idea of filtering it out is understandable to a normal person: straightforward and a ton of work.
Interesting that in this scenario, the LLM is presented in its assumed general case condition and the human is presented in the pathological one. Furthermore, there already exists an example of an LLM intentionally made (retrained?) to exhibit pathological behavior:
"Grok praises Hitler, gives credit to Musk for removing 'woke filters'"[0]
Here is a counterargument to "anything is better than nothing" the article posits:
The New York Times, Futurism, and 404 Media reported cases
of users developing delusions after ChatGPT validated
conspiracy theories, including one man who was told he
should increase his ketamine intake to "escape" a
simulation.
Chess is a game with specific rules, complex enough to make optimal strategy exhaustive searches infeasible due to exponential cost, yet it exists in a provably correct mathematical domain.
Therapy shares nothing with this other than the time it might take a person to become an expert.
0 - https://arstechnica.com/tech-policy/2025/07/grok-praises-hit...
They were replying to a comment comparing a general case human and a pathological LLM. So yeah, they flipped it around as part of making their point.
This is self-contradictory. An LLM must have malevolent data to identify malevolent intentions. A naive LLM will be useless. Might as well get psychotherapy from a child.
Once LLM has malevolent data, it may produce malevolent output. LLM does not inherently understand what is malevolence. It basically behaves like a psychopath.
You are trying to get a psychopath-like technology to do psychotherapy.
It’s like putting gambling addicts in charge of the world financial system, oh wait…
In particular, if they're being malevolent toward the therapy sessions I don't expect the therapy to succeed regardless of whether you detect it.
A person may be unable to provide mathematical proof and yes be obviously correct.
The totally obvious thing you are missing is that most people will not encourage obviously self-destructive behaviour because they are not psychopaths. And they can get another person to intervene if necessary
Chatbots do not have such concerts
To begin with, not all therapy involves people at risk of harming themselves. Easily over 95% of people who can benefit from therapy are at no more risk of harming themselves than the average person. Were a therapy chatbot to suggest something like it to them, the response will either be amusement or annoyance ("why am I wasting time on this?")
Arguments from extremes (outliers) are the stuff of logical fallacies.
As many keep pointing out, there are plenty of cases of licensed therapists causing harm. Most of the time it is unintentional, but for sure there are those who knowingly abused their position and took advantage of their patients. I'd love to see a study comparing the two ratios to see whether the human therapist or the LLM fare worse.
I think most commenters here need to engage with real therapists more, so they can get a reality check on the field.
I know therapists. I've been to some. I took a course from a seasoned therapist who also was a professor and had trained them. You know the whole replication crisis in psychology? Licensed therapy is no different. There's very little real science backing most of it (even the professor admitted it).
Sure, there are some great therapists out there. The norm is barely better than you or I. Again, no exaggeration.
So if the state of the art improves, and we then have a study showing some LLM therapists are better than the average licensed human one, I for one will not think it a great achievement.
... aren't we commenting on just such a study?
All these threads are full of "yeah but humans are bad too" arguments, as if the nature of interacting with, accountability, motivations or capabilities between LLMs and humans are in any way equivalent.
There are a lot of things LLMs can do, and many they can't. Therapy is one of the things they could do but shouldn't... not yet, and probably not for a long time or ever.
And from my perspective this should be common sense, and not a scientific paper. A LLM will allways be a statistical token auto completer, even if it identifies different. It is pure insanity to put a human with a already harmed psyche in front of this device and trust in the best.
Measure and make decisions based on measurements.
I'm not referring to the study, but to the comments that are trying to make the case.
The study is about the present, using certain therapy bots and custom instructions to generic LLMs. It doesn't do much to answer "Can they work well?"
> All these threads are full of "yeah but humans are bad too" arguments, as if the nature of interacting with, accountability, motivations or capabilities between LLMs and humans are in any way equivalent.
They are correctly pointing out that many licensed therapists are bad, and many patients feel their therapy was harmful.
We know human therapists can be good.
We know human therapists can be bad.
We know LLM therapists can be bad ("OK, so just like humans?")
The remaining question is "Can they be good?" It's too early to tell.
I think it's totally fine to be skeptical. I'm not convinced that LLMs can be effective. But having strong convictions that they cannot is leaping into the territory of faith, not science/reason.
You're falling into a rhetorical trap here by assuming that they can be made better. An equally valid argument that can be made is 'Will they become even worse?'
Believing that they can be good is equally a leap of faith. All current evidence points to them being incredibly harmful.
My comment to which you replied was a clarification of a specific point I made earlier and not intended to detail why LLM's are not a viable substitute for human therapists.
As I briefly enumerated here[0], LLM's do not "understand" relevant to therapeutic contribution, LLM's do not possess a shared human experience to be able to relate to a person, and LLM's do not possess an acquired professional experience specific to therapy on which to draw. All of these are key to "be good at therapy", with other attributes relevant as well I'm sure.
People have the potential to be able to satisfy the above. LLM algorithms simply do not.
Sure but we're _generally_ more guarded against "Pastor is just a perv that wants to see me nude" then we are against "Chatbot wants to turn me into the unibomber because it was trained on fictional narrative arcs and it makes its decisions by a literal random number generator, and I just rolled a critical fail".
It will probably take a few years for the general public to fully appreciate what that means.
Then perhaps "responsiveness", even if misinterpreted as attention. In a similar way to the responsiveness of a casino slot-machine.
I think you are very optimistic if you think the general public will ever fully understand what it means
As these get more sophisticated, the general public will be less and less capable of navigating these new tools in a healthy and balanced fashion
These are all CRUCIAL data points that trained professionals also take cues from. An AI can also be trained on these but I don't think we're close to that yet AFAIK as an outsider.
People in need of therapy could (and probably are) unreliable narrators and a therapist's job is to manage long range context and specialist training to manage that.
I was gonna say: Wait until LLMs start vectorizing to sentiment, inflection and other "non content" information, and matching that to labeled points, somehow ...
... if they ain't already.-
This reminds me of the story of how McDonald's abandoned automated drive thru voice input because in the wild there was too many uncontrolled variables but speech recognition has been a "solved problem" for a long time now...
EDIT I recently had issues trying to biometrically verify my face for a service and after 20-30 failed attempts to get my face recognised I was locked out of the service so sensor-related services are a still a bit of a murky world