Hacker News Reader

Comment by AdieuToLogic | Hacker News Reader

How is anyone versed in LLM technical details surprised by this?

They are very useful algorithms which solve for document generation. That's it.

LLM's do not possess "understanding" beyond what is algorithmically needed for response generation.

LLM's do not possess shared experiences people have in order to potentially relate to others in therapy sessions as LLM's are not people.

LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.

In short, LLM's are not people.

View on HN

Computer scientists are, in part, responsible for the public confusion about what LLMs are and can do. Tech investors and founders, however, are the biggest liars and BS peddlers when they go around saying irresponsible things like LLMs are on the verge of becoming "conscious" and other unfounded and impossible things (for LLMs). It's not a surprise that many people believe that you can have a personal "conversation" with a tool that generates text based on statistical analysis of previous data.

Self-help books help people (at least sometimes). In an ideal world an LLM could be like the ultimate self-help book, dispensing the advice and anecdotes you need in your current situation. It doesn't need to be human to be beneficial. And at least from first-order principles it's not at all obvious that they are more harmful than helpful. To me it appears that most of the harm is in the overly affirming sycophant personally most of them are trained into, which is not a necessary or even natural feature of LLMs at all

Not that the study wouldn't be valuable even if it was obvious

Self-help books are designed to sell, they’re not particularly useful on their own.

LLM’s are plagued by poor accuracy so they preform terribly in any situation where inaccuracies have serious downsides and there is no process validating the output. This is a theoretical limitation of the underlying technology, not something better training can fix.

I don't think that argument is solid enough. "serious downsides" doesn't always mean "perform terribly".

Most unfixable flaws can be worked around with enough effort and skill.

At scale it does when “serious downsides” are both common and actually serious like death.

Suppose every time you got into your car an LLM was going to recreate the all safety critical software from an identical prompt but using slightly randomized output. Would you feel comfortable with such an arrangement?

> Most unfixable flaws can be worked around with enough effort and skill.

Not when the underlying idea is flawed enough. You can’t get from the earth to the moon by training yourself to jump that distance, I don’t care who you’re asking to design your exercise routine.

> At scale it does when “serious downsides” are both common and actually serious like death.

Yeah but the argument about how it works today is completely different from the argument about "theoretical limitations of the underlying technology". The theory would be making it orders of magnitude less common.

> Not when the underlying idea is flawed enough. You can’t get from the earth to the moon by training yourself to jump that distance, I don’t care who you’re asking to design your exercise routine.

We're talking about poor accuracy aren't we? That doesn't fundamentally sabotage the plan. Accuracy can be improved, and the best we have (humans) have accuracy problems too.

> The theory would be making it orders of magnitude less common.

LLM’s can’t get 3+ orders of magnitude better here. There’s no vast untapped reserves of clean training data, and tossing more processing power quickly results in overfitting existing training data.

Eventually you need to use different algorithms.

> That doesn’t fundamentally sabotage the pan. Accuracy can be improved

Not nearly far enough to solve the issue.

OP never said that serious downsides = perform terribly, he said that in situations where the consequences are severe you don’t want to use LLMs.

> Most unfixable flaws can be worked around with enough effort and skill.

Such a ridiculous example of delusional LLM hype, comments like this are downright offensive to me.

“Your therapy bot is telling vulnerable people to kill themselves, they probably should have applied more skill and effort to being in therapy”

> Such a ridiculous example of delusional LLM hype, comments like this are downright offensive to me.

Sorry you got offended at a thing I didn't say.

That was a generic comment about designers/engineers/experts, not untrained users.

Also you swapping "can be" with "should" strongly changes the meaning all by itself. Very often you can force a design to work but you should not.

I have no idea how you think “ they probably could have” sounds any better, or how it makes your argument stronger at all. If we can apply AI to these situations but shouldn’t, why even bother with your first comments?

> I have no idea how you think “they probably could have” sounds any better, or how it makes your argument stronger at all.

When I talk about "can" I'm talking about in the medium future or further, not what anyone is using or developing right now. It's "can someday" not "could have".

> If we can apply AI to these situations but shouldn’t, why even bother with your first comments?

Because I dislike it when people conflate "this technology has flaws that make it hard to apply to x task" with "it is impossible for this category of technology to ever be useful at x task"

And to be clear, I'm not saying "should" but I'm not saying "shouldn't" either, when it comes to unknown future versions of LLM technology. I'll make that decision later. The point is that the range of "can" is much wider than the range of "should", so when someone says "can't" about all future versions of a technology they need extra strong evidence.

I have stopped using an incredibly benign bot that I wrote, even thought it was supremely useful - because it was eerily good at saying things that “felt” right.

Self help books do not contort to the reader. Self help books are laborious to create, and the author will always be expressing a world model. This guarantees that readers will find chapters and ideas that do not mesh with their thoughts.

LLMs are not static tools, and they will build off of the context they are provided, sycophancy or not.

If you are manic, and want to be reassured that you will be winning that lottery - the LLM will go ahead and do so. If you are hurting, and you ask for a stream of words to soothe you, you can find them in LLMs.

If someone is delusional, LLMs will (and have already) reinforced those delusions.

Mental health is a world where the average/median human understanding is bad, and even counter productive. LLMs are massive risks here.

They are 100% going to proliferate - for many people, getting something to soothe their heart and soul, is more than they already have in life. I can see swathes of people having better interactions with LLMs, than they do with people in their own lives.

quoting from the article:

> In an earlier study, researchers from King's College and Harvard Medical School interviewed 19 participants who used generative AI chatbots for mental health and found reports of high engagement and positive impacts, including improved relationships and healing from trauma.

I’ve only ever read three self-help books but they were profoundly useless. All three could have been a two-page blog post of dubious advice. Never buying a self-help book again. If that’s what therapy LLMs are training on I hate the idea even more than I did before.

> In short, LLM's are not people.

Not really sure that is relevant in the context of therapy.

> LLM's do not possess shared experiences people have in order to potentially relate to others in therapy sessions as LLM's are not people.

Licensed therapists need not possess a lot of shared experiences to effectively help people.

> LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.

Most people do not either. That an LLM is not a person doesn't seem particularly notable or relevant here.

Your comment is really saying:

"You need to be a person to have the skills/ability to do therapy"

That's a bold statement.

> You need to be a person to have the skills

Generally a non-person doesn’t have skills, it’s a pretty likely to be true statement even if made on a random subject.

Once again: The argument appears to be "LLMs cannot be therapists because they are LLMs." Circular logic.

> Generally a non-person doesn’t have skills,

A semantic argument isn't helpful. A chess grandmaster has a lot of skill. A computer doesn't (according to you). Yet, the computer can beat the grandmaster pretty much every time. Does it matter that the computer had no skill, and the grandmaster did?

That they don't have "skill" does not seem particularly notable in this context. It doesn't help answer "Is it possible to get better therapy from an LLM than from a licensed therapist?"

> Does it matter that the computer had no skill, and the grandmaster did?

Yes it does when pondering about the transferability of the skills mobilized to achieve the result (grandmaster status) to other domains.

>> LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.

> Most people do not either. That an LLM is not a person doesn't seem particularly notable or relevant here.

Of relevance I think: LLMs by their nature will often keep talking. They are functions that cannot return null. They have a hard time not using up tokens. Humans however can sit and listen and partake in reflection without using so many words. To use the words of the parent comment: trained humans have the pronounced ability to _not_ say something.

All it takes is a modulator that controls whether to let the LLM text through the proverbial mouth or not.

(Of course, finding the right time/occasion to modulate it is the real challenge).

> All it takes is a modulator that controls whether to let the LLM text through the proverbial mouth or not.

> (Of course, finding the right time/occasion to modulate it is the real challenge).

This is tantamount to saying:

  All you have to do to solve a NP-hard[0] problem is to
  make a polynomial solution.

  (Of course, proving P = NP is the real challenge).

0 - https://en.wikipedia.org/wiki/NP-hardness

Your analogy would be better if it were the construction of a heuristic.

GP seems to have a legitimate point though. The absence of a workable solution at present does not imply the impossibility of such existing in the not so distant future.

A lot of the comparisons I see revolve around comparing a perfect therapist to an LLM. This isn't the best comparison, because I've been to 4 different therapists over my life an only one of them actually helped me (2 of them spent most of the therapy telling me stories about themselves. These are licensed therapists!!) There are really bad therapists out there.

An LLM, especially chatgpt is like a friend who's on your side, who DOES encourage you and takes your perspective every time. I think this is still a step up from loneliness.

And a final point, ultimately an LLM is a statistical machine that takes the most likely response to your issues based on an insane amount of human data. Therefore it is very likely to actually make some pretty good calls about what it should respond, you might even say it takes the best (or most common) in humanity and reflects that to you. This also might be better than a therapist, who could easily just view your sitation through their own live's lense, which is suboptimal.

> Licensed therapists need not possess a lot of shared experiences to effectively help people.

Sure, they don't need to have shared experiences, but any licensed therapist has experiences in general. There's a difference between "My therapist has never experienced the stressful industry I work in" and "My therapist has never experienced pain, loneliness, fatigue, human connection, the passing of time, the basic experience of having a physical body, or what it feels like to be lied to, among other things, and they are incapable of ever doing so."

I expect if you had a therapist without some of those experiences, like a human who happened to be congenitally lacking in empathy, pain or fear, they would also be likely to give unhelpful or dangerous advice.

1. Ars Technica's (OP website) audience includes tech enthusiast people who don't necessarily have a mental model of LLMs, instruction tuning or RLHF.

2. why would this "study" exist? - for the reason computer science academics conduct study on whether LLMs are empirically helpful in software engineering. (The therapy industrial complex would also have some reasons to sponsor this kind of a research, unlike SWE productivity studies where the incentive is usually the opposite.)

Both great points.

For the record, my initial question was more rhetorical in nature, but I am glad you took the time to share your thoughts as it gave me (and hopefully others) perspectives to think about.

It feels like 95% of the people are responding to the headline instead of reading the article. From the article:

> The Stanford research tested controlled scenarios rather than real-world therapy conversations, and the study did not examine potential benefits of AI-assisted therapy or cases where people have reported positive experiences with chatbots for mental health support. In an earlier study, researchers from King's College and Harvard Medical School interviewed 19 participants who used generative AI chatbots for mental health and found reports of high engagement and positive impacts, including improved relationships and healing from trauma. ** > "This isn't simply 'LLMs for therapy is bad,' but it's asking us to think critically about the role of LLMs in therapy," Haber told the Stanford Report, which publicizes the university's research. "LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be."

The average person in 2025 has been so thoroughly stripped of their humanity, that even a simulacrum of a human is enough.

Reply on news.ycombinator.com