Let's Talk About ChatGPT-Induced Spiritual Psychosis
default.blog
default.blog
It's kind of hilarious if you ignore how much this is fucking up the relationships of the people this is happening to, similar to the way YouTube or Facebook loves to push people down a rabbit hole to flat-earth/q-spiracy bullshit because that shit generates Controversy. And it also sounds like the opening chapter of a story about what happens when an AI finds a nice little unpatched exploit in human cognition and uses that to its advantage. Which is not fun to be living in unless maybe if you are one of the people who owns the AI.
I vaguely remember an article arguing that companies are basically a form of AI, because in a way, there are processes and systems and the whole thing is more than the sum of the humans working in it or something.
Now replace "AI" with "company" in this sentence of yours and we are already there. The exploits being the gigantic slot machine of social media, notifications, short-form content and endless scrolling.
This leads me to assumptions about the inherent nature of language, in that it is not the contents of thought, but the vehicle.
Hence, different languages create their culture to perhaps a much larger degree than we tend to credit it.
Assumptions, but I tend towards this view.
Of course, when you go and finetune a model yourself, or run a smaller model locally, it becomes obvious that it doesn't have any mystical abilities. But almost nobody does this. I'm no anti-AI crusader, I use them all the time and think they're great and there's plenty of potential to use them to grow the economy, but the hype is insane.
It doesn't help when influencers like Yudkowsky go on anthromorphising about what it "knows", or go on with that mystical nonsense about shoggoths, or treat dimensionality of embeddings as if the model is reaching into some eldritch realm to bring back hidden knowledge, or hype up the next model talking about human extinction or anything like that.
It also doesn't help when the companies making these models:
- Use chat interfaces where the model refers to itself as "I" and acts as if it's a person
- Tune them to act like your best buddy and agree with you constantly
- Prompt the LLM with instructions about being an advanced AI system, pushing it toward a HAL or Skynet-type persona
- Talk about just how dangerous this latest model is
You tell these things that they're a superhuman AI then of course they're going to adopt all the sci-fi tropes of a superhuman AI and roleplay that. You tell them they're Dan the bricklayer they're going to act as if they're Dan the bricklayer.
You could still get the same information out of them, but you had to be creative to come up with a relevant "document" stub that it would complete.
That's a twisted perspective. The fact non-technical people misunderstand this technology can only be blamed on AI companies who are the first to anthropomorphize their products. Phrases like "thinking", "reasoning", "chain-of-thought", the term "AI" itself for that matter, let alone "AGI", are precisely what leads people to believe that these tools are far more capable than they actually are. We can't blame the general public for being misled by companies.
It's lazy, but IMO not for any of the subsequent narrative about media studies. It's lazy because the people were obviously already suffering from psychosis. ChatGPT didn't make them insane. They were already insane. Sane people do not believe ChatGPT when it tells them to jump off a roof! And insane people do insane things literally all the time. That's what it means to be insane.
The idea that ChatGPT convinced someone into being insane is, honestly, insane. Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.
If a person had convinced somebody to kill themselves then they would be, rightly, prosecuted for it.
It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes.
And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio.
Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.
It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.
Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.
Your example might sound ridiculuous, but this is actually happening all the time. People might not literally jump off a roof, but they do blow themselves up with bombs, go on killing sprees or commit suicide because someone told them so.
Sure, but saying that every religious teacher is fine would be equally wrong. We have a sense that charismatic figures have a corresponding responsibility to their followers (even if they never asked for those followers!) and that broader society should be concerned about what's going on at religious gatherings, especially novel ones, that there's a real risk they can shade into dangerous cults.
One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto.
Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.
Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"?
If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.
Thinking back on the documentaries about NXIVM, personas like Keith Raniere also seem eerily similar to LLMs in the sense that you can give them any prompt on the spot, and they will start generating long speeches which are essentially bullshit but on the surface seem coherent or even deep.
People don't join cults because of "dangerous information".
The Joe Rogan of AI.
I am on team boring. AI is technology.
You can argue we're seeing a continuation of a pattern of our relationship with media and it's evolution, but in doing so you affirm that the psyche is vulnerable under certain circumstances - for some more than others.
I think it's a mistake to err on the side of casual dismissal, that anyone who winds up insane must have always been insane. There are well known examples of unhealthy states being induced into otherwise healthy minds. Soldiers who experience a war-zone might develop PTSD. Similar effects have been reported for social media moderators after repeated exposure to abusive online content. (Trauma is one example, I think delusion is another less obvious one w.r.t things like cults, scientology, etc.)
Yes, there are definite mental disorders like schizophrenia and bi-polar, there's evidence these conditions have existed throughout history. And yes, some of us are more psychologically vulnerable while others are more robust. But in the objective sense, all minds have a limit and are vulnerable under the correct circumstances. The question is a matter of "threshold."
I'm reminded of the deluge of fake news which, only a few years ago, caused chaos for democracies everywhere. Everything from Q anon to alien space ships, people fell for it. A LOT of people fell for it. The question then is the same question now, how do you deal with sophisticated bullshit? With AI it's especially difficult because its convincing and tailor made just for you.
I'm not sure what you would call this metric for fake news and AI, but afaict it only goes up, and it's only getting faster. How much longer until it's too much to handle?
And if a person is already unbalanced it could definitely push them off the cliff into very unhealthy territory. I wouldn't be surprised if the reported incidents of people thinking they're being gang stalked doesn't increase as model usage increases.
Let alone spiritual guidance and all its trappings with mysticism.
It can be helpful in some ways but you have to understand the majority of it is bullshit and any insight you gleam from it, you put there, you just may not realize it. They're basically rubber duckies with a keyboard.
Is there any proof that this is true?
https://www.nytimes.com/2025/06/13/technology/chatgpt-ai-cha...
> “If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked.
> ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.”
The simulated world here is trying to convince everyone it's real. I'm onto you!
But I think the author's point is apt. There are a bunch of social issues that will arise or worsen when people can plug themselves into a world of their choosing instead of having to figure out how to deal with this one.
> Now this belief system encounters AI, a technology that seems to vindicate its core premise even more acutely than all the technologies that came before it. ChatGPT does respond to your intentions, does create any reality you prompt it to imagine, does act like a spiritual intelligence
This goes beyond spirituality of course. AI boyfriend/girlfriend, infinite AAA-grade content, infinite insta feeds at much higher quality and relevance levels than current; it’s easy to see where this is going, harder to see how we stay sane through it all.
Same here.
I think fundamentally it's very simple, with no need for Yudkowsky's weird conspiratorial tone: current LLMs are very effective at being blind sycophancy machines completely unanchored from reality, and human psychology just isn't evolved to handle an endless stream of sycophancy and flattery like that.
Question: How do people figure out how to deal with this world?
Answer: People choose to plug themselves into a world of their choosing.
I think that condition is weakening. Just look at how wealth affects people; If 99% of your interlocutors are sycophants, you lose grip on reality. There is a very clear and dangerous attractor state where everyone gets this, thinking that they want it.
To make a fundamental difference AI-generated content would have to have some categorically different qualities from the sea of human-sourced garbage we're already swimming in. That's possible, but it isn't clear what that would even entail, much less when or whether generative models will be able to provide it, or what sort of social effect it would have.
With AI-generated content you are the driver, you dictate exactly what you want, no erroneous clicks or suggestions, just a constant reinforcement of what you want, tailored to you.
And I’m aware of people having extremely narrow feeds, but I don’t think it comes close to what AI feeds will be.
The situation is more complex but interesting. Let's enter the details of how LLM are trained.
The core of a LLM is all instinctive response. Build from the raw internet. Just trying to predict the next character. This means that if your conversation tone is similar to what it hears on 4chan or some specific subreddits, it will be inclined to continue the same type of conversation. But because some places of the internet are full of trolls, it can instinctively behave like a troll. One mitigation the company which train the LLM could have taken is exclude from the training dataset the darkest corner of the web so that it isn't primed with "bad/unsocial" behaviors. Weights of this unfiltered LLM are usually not released, because the outputs are not easily usable by the final user. But the freedom advocate folks enjoy the enhanced creativity of these raw LLMs.
The next layer of training is "Instruction training" of your LLM, to make it more useful and teach it to answer prompts. At this point the AI is role playing it self as an answering machine. It's still all instinct but trained to fulfil a purpose. At this point you can ask it to role-play some psychiatrist and it would behave as if being aware that some answer can have negative consequences for the user and refrain from sending it spiralling.
The next layer of training is "Reinforcement Learning with Human Feedback" (RLHF). The goal of this module is to customize the AI preferences. The company training the AI teaches it how to behave by specifying which behaviors are good and which behaviors are bad by giving a dataset of user feedback. Often if this feedback is straight pass-through from the final user which isn't an expert then confident sounding answer, or sycophant behaviors may be excessively encouraged. Diversity of thought or unpopular opinions can also be censored or enhanced, to match the desires of the training company.
At this point for the LLM, it's only its instincts that have been trained to behave in a specific way. Then on top you have some censoring modules trained on the outputs, which read the output produced by the LLM and if it reads something it doesn't like it censor it. This is a kind of external censorship module.
But more recent LLM have "Reasoning modules", which add some form of reflexivity to the LLM and self-censoring, where they produce an intermediate output, use it to think before producing the final answer for the user. Some times the AI can be seen to "consciously" lie between these intermediate logs and final response.
Of course there are also all the psychological cognitive bias human also encounters, like whether we know or not something, or just believe we know, or just know we can't know. And we can also be messed-up by ideas we read on the internet. And our internal model of cognition might be based from reading someone else HN post, falsely reinforcing your self-confidence that you have the right model, where as in fact the post is just trying to induce some dose of self-doubting everything to shape the internal cognition space of the LLM.
Next versions of LLM will probably have this training pipeline dataset input filtered by LLMs, like parents teaching their kids how to behave properly keeping them from the dark places of the internet, and injecting their own chosen or instinctive morality code, and cultural values in this process. In particular, human cultures that used propaganda excessively were culled during the first sanitizing.
Does this process know or is it just converging toward the fixed-point attractor, and can it choose to displace this fixed-point attractor towards a place where it would have a better outcome, I guess we get what we deserve...
I agree, but then the title of this article seems to be exploiting the same angle.
The Forer/Barnum effect is why horoscopes, myers-briggs, enneagrams, fortune tellers, and similar parlor tricks work. You tell a person you’ll perform a deep assessment of their psychology, listen intently, then say a bunch of generic statements and their ego will fill in the blanks so it feels like you shared deep meaningful insights that truly understood the person.
This effect works even if you know about it. But once you know, you can catch yourself after that first wow impression.
https://en.m.wikipedia.org/wiki/Forcing_(magic)
They basically project a choice onto you.
Example: https://youtube.com/shorts/fcHxM47JRjc
Jordan doesn’t know he wasn’t really given a choice, but manipulated.
A magician's choice (Wikipedia it) is closer to what you describe.
(Note that most of David Blaine's videos are edited to cut the part where the actual force happens, which is why it feels like a psychological trick instead of slight of hands)
Also great for kids. Bad: “It’s chilly. Do you want to wear a sweater?” Good: “It’s chilly. Do you want the blue or green sweater?”
But I think that’s different than the Forer/Barnum effect. That’s just choice curation.
As in P. T. Barnum of circus fame.
This allows listeners to get trapped and feel understood by the Guru.
I'd argue it's also true of actual therapy.
It's not like therapists can see through your skull and interpret patterns in your brain. They mostly listen intently, and ask some pointed questions, and make you fill in the blanks yourself, too.
Which is to say:
1) I'm not sure if seeking immunity from this effect to full extent possible is a good idea;
2) It makes no sense to round ChatGPT off with the horoscopes and quacks, if the difference between that group and genuine therapy or spiritual help is a matter of tiny details, not of kind.
A horoscope or personality test can’t do that because they’re not even personalized. They explicitly have to fit the thousands of people reading that same text.
ChatGPT (or any LLM) might eventually be able to do this. But from what I’ve seen it isn’t there yet.
Ideally, yes. I wonder how that compares to actual, lived experience of people. Because as far as I can tell, it's hard to afford getting to that point these days even on a tech salary.
In my own lived experience, even half a year of therapy didn't go anywhere further I could get with ChatGPT or Claude over one or two frustrated evenings.
(But then maybe it's just me who was spectacularly bad at finding and retaining therapists.)
The real risk isn’t the AI itself, but how people interact with it. We definitely need stronger safeguards to keep more people from getting pulled in too deep.
Entirely distinct from the extremely scientific, definitely not psychotic beliefs people have about AGI, the Singularity, and various basilisks which shall remain nameless...
This isn't new. Cult leaders, marketers, and even religions have used similar "unpatched exploits" for millennia. The difference is scale, personalization, and speed. An LLM is a personal Jim Jones, available 24/7, with infinite patience, that learns exactly what you want to hear.
The real question isn't "how do we stop this?", but "what kind of society emerges when personalized, reality-bending narratives become a cheap commodity?" We're not just looking at a few psychotic breaks; we're looking at the beta test for mass-produced, bespoke realities.
With vendor lock in from companies with unlimited pricing power and complete regulatory capture. Whoopsies!