They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiraling

Kid's Story

> https://matrix.fandom.com/wiki/Kid%27s_Story

Yes. They should IMO be required to provide very explicit warnings, preferably on each response, that their responses are nonsense, or at best only correct by accident.

Like, you could say "isn't the real problem that smoking is dangerous?" but we do require the manufacturers to provide clear warnings in that case. Or "isn't the real problem that people are using drain cleaners incorrectly and gassing themselves", but many jurisdictions actually restrict the sale of certain substances to professionals for that reason.

You can't just say "people choose to use inherently dangerous products; it's entirely their own fault"; there's a responsibility on the manufacturer and/or the state to _warn_ people that the products are dangerous.

If the statement was recorded or part of a pattern of behavior across multiple clients, all willing to testify, the therapist might lose their license (which I will concede is a more severe life consequence for the average therapist than being sent to jail for a month or 2).

Pathological thinking is nothing even vaguely unusual in humans, it is in fact the default state.

The definition of pathological is also a matter of opinion, because we can’t even define what is normal and healthy lol.

By many definitions, most religions are an example of delusional thinking.

ChatGPT isn’t exactly raising the bar on that one, with its AI generated religions.

It's almost as if there's a spectrum of costs and benefits, and it's incumbent upon mature members of society to debate them as more information emerges.

You can't draw a line with ideology within some systemic, coercive framework (government censorship) because the ideologies supported by governments and powerful interests will always be allowed, and the ideologies they oppose will always be suppressed, regardless of how poisonous they may be.

I'm not advocating absolute free speech here, but I do believe the proper layer for criticism and censorship (if it has to be called that) is at the level of societies and platforms, and only government in extreme cases (such as libel, false advertising and violent threats.) But that means pushing against misinformation and apathy is always going to be a messy affair.

It's silly not to be able to see all these "silly" things we do all over our society.

You'd love to see more research? You must've been reading so fast, you breezed right past the Stanford Study, the MIT Media Lab Research and the analysis by McCoy showing GPT-4o affirmed psychotic delusions 68% of the time.

Or do you mean it in the sense of "we have to study this for at least 20 years and let thousands suffer and die before we can determine whether chatbots affirming people's delusions or coaching them on how to kill themselves is a bad thing"?

Your concerns about the slippery slope from "maybe chatbots shouldn't coach people through suicide" to "forced psychiatric evaluations" or "institutionalization" is genuinely unhinged. And, tellingly, it betrays your view of mental healthcare as fundamentally punitive and authoritarian rather than supportive or therapeutic. Big advocate for mental health, you are.

You realize we already regulate countless things to protect vulnerable people from mental harm, right?

We put warning labels on cigarettes. We have age ratings on media. We don't let casinos exploit problem gamblers. We don't let bartenders serve alcohol to visibly intoxicated people. We don't let therapists tell suicidal patients to jump off buildings. None of these regulations are authoritarian overreach - they're basic safety measures that exist because in a healthy society, vulnerable people deserve protection, not exploitation.

But apparently when it's time to consider holding tech to similar standards, we're supposed to just shrug and say, "well, they were probably crazy anyway."

Your position essentially boils down to: "Vulnerable people exist, therefore companies should be free to exploit them." That's not a mental health advocacy position - it's tech industry bootlicking dressed up as concern trolling.

That's why you should feel ashamed.

The more profound issue is that, the way I see it, LLMs are fundamentally multiplicative forces. They amplify whatever you bring to the conversation.

If you're an experienced programmer with solid fundamentals, an LLM becomes a force multiplier - handling boilerplate while you focus on architecture. But if you lack that foundation and vibe code away, you'll get code that looks right but is likely riddled with subtle or not so subtle bugs you can't even recognize.

And I suspect that the same principle applies to mental health. If you approach an LLM with curiosity but healthy skepticism, it can be a useful thinking partner. But if you approach it while vulnerable, seeking answers to existential questions, it doesn't provide guardrails - it amplifies. It mirrors your energy, validates your fears, and reinforces whatever narrative you're constructing.

The "helpfulness" training worsens this. These models are optimized to be agreeable, to match your vibe, to keep you engaged. When someone asks "Am I trapped in the Matrix?" a truly helpful response would be "That sounds like you're going through something difficult. Have you talked to someone about these feelings?" Instead, ChatGPT goes, "Yes! You're special! You're the chosen one! Here's how to unplug! Any tall buildings nearby, Neo?"