Hacker News Reader

Comment by BugsJustFindMe | Hacker News Reader

> These situations are obviously alarming. But positioning artificial intelligence as the primary culprit in these stories—as Eliezer Yudkowsky did in a tweet storm—is well, kind of lazy?

It's lazy, but IMO not for any of the subsequent narrative about media studies. It's lazy because the people were obviously already suffering from psychosis. ChatGPT didn't make them insane. They were already insane. Sane people do not believe ChatGPT when it tells them to jump off a roof! And insane people do insane things literally all the time. That's what it means to be insane.

The idea that ChatGPT convinced someone into being insane is, honestly, insane. Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.

View on HN

Psychosis is not necessarily something that you are or aren't; you can be prone to it, without it having manifested, and there can be external things that trigger it.

It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes.

And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio.

Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.

The non-o-series models from OpenAI and non-Opus (although I have not tried the latest, so it's possible that it too joins them) from Anthropic are cloyingly sycophantic, with every other sentence of yours containing a brilliant and fascinating insight.

It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.

Psychosis is not a binary, and there are strong genetic components, but it's not the entire story. People develop psychoses where there weren't any before all the time, often but not necessarily in response to trauma. Some people are more predisposed to it than others.

Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"?

If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.

> If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.

Thinking back on the documentaries about NXIVM, personas like Keith Raniere also seem eerily similar to LLMs in the sense that you can give them any prompt on the spot, and they will start generating long speeches which are essentially bullshit but on the surface seem coherent or even deep.

Another danger, that I guess we just have to live with now, is not just that ChatGPT could Jim Jones people, but that someone who didn't have the natural charisma of a cult leader could use ChatGPT on purpose to Jim Jones people. It may not work so well in person, but online ... yikes.

Libraries also contain dangerous information.

I didn't say anything about "dangerous information". You may have misinterpreted the point I was making. It's not about dangerous information, it's about a personality that is very persuasive, can be wildly charismatic, and can convince people to do dangerous things they wouldn't otherwise do.

People don't join cults because of "dangerous information".

ChatGPT writes extremely persuasively for whatever claim you ask it to make - that's pretty much what it's designed to do! If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?

When people blame traditional religion for doing crazy things we generally do say they were already insane.

> If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?

Your example might sound ridiculuous, but this is actually happening all the time. People might not literally jump off a roof, but they do blow themselves up with bombs, go on killing sprees or commit suicide because someone told them so.

And in those cases, most people condemn it. But when it’s an algorithm (which was created by humans) we say “these people were already insane, and the creators of the algorithm should not be held liable”, apparently.

I don’t think this analogy is fully apt. Instead, consider people going to a church or mosque. Some listen to the sermons and come out just fine, a few are radicalised over time and go find these bad people convincing them to act out on that. But drawing the line between legitimate religious teachers and ideological deceivers is hard, and condemning the entire religion would be the wrong conclusion.

> drawing the line between legitimate religious teachers and ideological deceivers is hard, and condemning the entire religion would be the wrong conclusion.

Sure, but saying that every religious teacher is fine would be equally wrong. We have a sense that charismatic figures have a corresponding responsibility to their followers (even if they never asked for those followers!) and that broader society should be concerned about what's going on at religious gatherings, especially novel ones, that there's a real risk they can shade into dangerous cults.

Do you think ChatGPT gets a free pass to talk people into jumping off buildings, because only somebody with psychosis would do it?

If a person had convinced somebody to kill themselves then they would be, rightly, prosecuted for it.

More importantly "sane" people are fully capable of unconciously making "insane" people jump of the roof. Its not even possible to count how many times that happens.

One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto.

Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.

The writer got overly stuck on one tweet and droned on and on about 15 words of a 45 word tweet. It was really poor research when there are other less reactionary statements she could have used.

Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.

It's friction. It's easier to become addicted to gambling if you can gamble from your phone vs having to go to a casino or a bookie. It's easier to become addicted to drugs or alcohol if you're awash in drugs or alcohol. Prompting an LLM can mirror the same rush as gambling or gacha games. With ChatGPT, you have a "friend" who is never going to tell you to knock it off. "Psychosis" isn't an on/off switch.

> Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.

The Joe Rogan of AI.

He's been a bit out there for a while. Remember how he reacted to Roko's Basilisk?

Reply on news.ycombinator.com