Can A.I. be blamed for a teen's suicide?
nytimes.com
nytimes.com
That said, I think this should throw a bucket of cold water on anyone recommending using generative AI as a therapist/counsellor/companion or creating and advertising "therapist" chatbots, because it simply isn't reasonable to expect them to respond appropriately to things like suicidal ideation. That isn't the purpose or design of the technology, and they can be pushed into agreeing with the user's statements fairly easily.
OTOH we have examples of real humans typing "DO IT FA*OT" on livestreams. ¯\_(ツ)_/¯
In the earlier chat session, being more explicit, the bot seemed to attempt to talk the user out of self-harming behaviour.
Crucially, Cr1TiKaL ran this test after an article was written about this phenomenon[2], where Character AI claimed "it has added a self-harm resource to its platform and they plan to implement new safety measures, including ones for users under the age of 18." Obviously the guard rails were not implemented if the chatbot in the news story was still gaslighting its users.
[1]: https://youtu.be/FExnXCEAe6k?t=4m7s
[2]: https://www.cbsnews.com/news/florida-mother-lawsuit-characte...
Reading through it, that's the opposite of what happened.
> it wasn't capable of detecting euphemistic references to suicide
Yeah, that's a key component too. Due to that lack of "understanding" (sic), it literally encouraged the kid. :( :(
Sorry, I may have been carrying knowledge over from other articles on the same incident that showed additional transcripts. It seems that when he explicitly referred to death and suicide, the bot discouraged him from doing so. When he referred to suicide using euphemistic terms such as "coming home", then it gave responses that were encouraging (because it did not detect that those statements were about suicide).
We probably just shouldn’t sell echo chambers to children, regardless of whether they are AI based or human.
With hindsight, or sufficient emotional intelligence and context about an individual’s life beyond the role play it may be possible to conclude that someone is at risk, but honestly I’m not even sure that a person doing this role play online would have necessarily figured it out.
Many people have destructive tendencies of one kind or the other. It's how you deal with them that matters as to whether they actually become destructive, and how much so.
Merely reducing this to "the root cause" is not really helpful – it's about whether this app contributed, and if so, by how much?
Suicide is of course the most tragic and extreme negative outcome, but one must also wonder about all the less extreme tragic and negative outcomes.
[0] https://www.nytimes.com/2024/10/23/technology/characterai-la...
[1] https://www.hsph.harvard.edu/means-matter/means-matter/youth...
Also, I doubt most medicine cabinets have lethal doses of drugs, but I don't have any data to back up that assumption.
If this person actually thought dying would allow them to connect with a chatbot, I don't think they should have easy access to a weapon, and suicide is only one of the reasons for that. This child is dead, and I see no reason to suspect the chatbot is directly involved in the reason, where the weapon is clearly directly involved. Given that's the method they chose, I'd hope we could take preventive measures to reduce needless deaths from guns. So then and only then we can start on needless deaths from overdose. I believe we should solve the largest problem first. Do more children die from guns or from an overdose?
Yours is an argument of the form: "He was stabbed by a knife. Do let's not talk about why, let's just make it harder to get a knife, since that is how he died."
Maybe gun access should be harder, maybe it should not. That is not relevant. The issue is that an impressionable youth was talked into suicide by an LLM and nobody stopped it! Not LLM vendors making money on the interaction, not governments who keep jabbering about protecting youth online, not parents who should have cared enough to take an interest. Even if you took away guns, knives, and letter openers, people would kill themselves (see those stats i mentioned). And from now on, we also have to worry that LLMs will keep talking them into it, unless the actual issue is fixed.
> [The] issue is that an impressionable youth was talked into suicide by an LLM and nobody stopped it.
Right, that is the issue. Except, I disagree there's enough evidence for a reasonable person to conclude that the chatbot talked him into shooting himself. Where as there is plenty of evidence that a child with symptoms of depression had uncontrolled access to a loaded firearm.
But to the strawman representation of my argument, let's talk about both?
Unless the chatbot suggested he should kill himself, I don't think the chatbot is the problem in this example. Did it? or did it just repeat a banal agreement to some non-violent fantasy? Then, if my assumption is correct, that the chatbot didn't try to talk him into it. What other steps are there that could have prevented this suicide?
Parenting and bans on all girlfriend/boyfriend like LLMs for under-18s, like we control access to other dangerous and mind altering substances and objects.
But hypothetically is this ban just for people under 18? What about the other risks? The vast majority of phishing targets are old generations? Would you say we should also ban access to LLMs to people over 55 as well? If someone could talked into suicide, they could probably also be talked into into drinking, shouldn't that ban on LLMs extend to 21?
As a gun owner, the original criticism is quite valid as a contributing cause. Firearms have been socialized by mass media as being complete toys, and the idea that it's reasonable to just have them sitting around your house where your kids can access them in their own developing child context is just nuts. And it's not really sustainable to be reflexively against the idea of every type of regulation, especially ones for setting responsible normative behavior.
We don’t know how much an LLM was actually involved. You can’t just talk someone normal into it, you have to drag their neck to that cliff. Talking only works for those already on the edge.
It could be yet another rejection or fall or a dead cat on the road. But it was an LLM chat. Oh, easy, that’s the culprit!
Why a person was on the edge in the first place interests no one. It’s a wicked game, we ignore them walking onto it, but when it comes to jumping, we are like nooo, wait, don’t. Put a fence and pat ourselves on the back - saved a life today. As if eternal suffering on the edge was the ultimate goal of life saving.
Women who "attempt suicide" are often sending out a cry for help. The people who half-heartedly try something need and want the attention. They want to make a mess and have people rush in and care for them. They want to put a punctuation mark on the drama that is occurring in their life and elicit some change.
The men who are shooting themselves just want to get out of their suffering and finish themselves off. They don't want more attention or more help. It's sometimes not easy to slink off and quietly commit suicide before someone intervenes. Someone who succeeds in this task is "dead serious" about dying, not summoning help, or causing more drama.
That's the whole reasoning behind the methods and the severity of attempts. Perhaps there is a gender divide, but only because of the differing motives and end goals.
Male suicide attempts are on average more serious and generally a lot more succesful. And that is with or without access to guns.
it's missing the point of the refutation. It's also needless demeaning. it's also wrong. As a rule, a suicide attempt is always to end or reduce their suffering. (even 'fake attempts' where the self harm behavior is primarily attention seeking with something obviously not lethal)
> Many suicide attempts, especially by women, are obviously not serious but rather a cry for help. This is well-known and obvious to anyone having experience with these patients.
This is only true when you are including 'apparent attempts, with actual attempts. The desire to die is often fleeting, if you choose a quick acting method such as guns, you're not given a chance to change your mind. If you choose to overdose, you're given the opportunity to reconsider, and because it's not instantly fatal, you're able to seek help.
This is well known and obvious to anyone working with these patients.
It's also taught in every single therapeutic psych class I've even had, that disregarding the experiences in the way you describe it is actively harmful, so I really hope you're not currently working with these patients if that's the most compassion and patience you're able to evoke.
> Male suicide attempts are on average more serious and generally a lot more succesful. And that is with or without access to guns.
This isn't something I've heard (but I'm also no longer in psych) do you have a citation I might be able to search for, (other than the obvious keywords?)
Let me ask you this. If you leave your wallet filled to the brim with cash on a park bench somewhere, do you think the likelyhood increases that it will get stolen, compared to if it’s in your pocket?
That you accept theft as a daily norm is a failure of your society already. It was not always so, and it need not be so. Suicide is the same. And the proper way to address it is not to put everyone in a padded room with no access to pills or guns or knives.
I am talking about statistics and probabilities. It is obvious that there is a higher risk that a suicidal person will kill themselves if there is a firearm close by. If the kid in questions father had locked away his gun, there isa big possibility he would still be alive.
Have a nice day
Define 'easy'. Most people who attempt to poison themselves fail.
Firearms instructor Claude Werner writes that “[if] you’re not willing to spend a little bit of time, money, and effort to keep firearms out of unauthorized hands, then get rid of your guns.” (1)
He’s done a lot of research and writing on negative outcomes like this one, and said it completely changed the way he views things - it certainly did for me.
I see lots of discussion about what gun and caliber to get, but the essential, potentially life saving, safety rules for living with guns are an afterthought - perhaps there’s not much money to be had as there is in selling a gun.
1: https://thetacticalprofessor.net/2016/01/24/serious-mistake-...
It's like the 1980s Judas Priest thing all over again.
Who's to blame?
My question here is, what the hell were the parents doing, not removing this obviously destructive intrusion in his life? This reads to me the same as if he had been using drugs but the parents didn’t take away his stash or his paraphernalia.
For the sake of your children, people, remember that a cellphone is not an unequivocal good, nor a human right that children are entitled to unlimited use of, and there are plenty of apps and mechanisms by which you can monitor or limit your child’s use or misuse of technology.
Also, just don’t give kids screens. Period. A laptop maybe if they are using it for creative purposes, but the vast majority of social media and consumption that is achieved by children on cellphones and tablets is a negative force in their lives.
I see 3 year olds scrolling TikTok in the store these days. It makes me ill. Those kids are sooooooo fucked. That should legit be considered child endangerment.
We contacted teachers about our child downloading and watching anime and playing games all day in school. They wouldn’t/couldn’t do anything. We requested that the school take the computer away and give hardcopy assignments. They refused because that would invite notice from other students which could lead to bullying. That’s what they told us. I found the acceptable computer use policy on the school website and tried playing that card. Turns out our child hadn’t actually even signed it the last year…but that didn’t actually matter, and the school didn’t enforce the policy anyway.
The schools here won’t actually discipline kids anymore. We would get emails from the principal begging parents to tell their kids that they’re not supposed to leave school grounds at lunch, but every day at least a hundred kids would just run out. (Our kid didn’t do this…I guess watching anime in the corner of the cafeteria prevents truancy…yay?)
The last two-and-a-half years of high school were so exhausting trying to find anything that would work. Nothing did. Two parents and a therapist trying to counter one teenager, bad family influences, the school system, and multibillion-dollar internet corporations that intentionally work to addict people is a very uneven situation.
If schools are going to provide these things, they should have the sites the kids might need to access white-listed and block everything else. Telling parents to try and block things is not realistic.
Other parents are going to fight that though because it's a free pacifier for their kids.
Parenting involves doing a lot of things that will probably be wildly unpopular with your children if you are taking your responsibilities seriously.
Remember, as a parent you have a specific role. That role is not to be a friend to your child, but rather to be a parent. Making these roles clear with everyone involved is what makes it possible to parent effectively and still be on good terms with your child.
Asking teachers to switch to paper assignments might not be feasible depending on the curriculum they're being asked to teach against, number of students in the classroom (I'm going to assume that your kid is in a public school; class sizes have been increasing due to funding cuts), books that they're using, etc.
For example: one of the classes my wife taught years ago was designed around an online LMS. All quizzes, learning checks, exams and practice was done on this platform. Some of her students didn't have access to a computer at home. They would either be given a loaner or were given instructions on how to obtain one from the local library. In this situation, switching to paper grading was not possible.
In reality, giving kids Doubleclick Chromebooks with unsupervised general Internet access is gross negligence on the part of the school. Pacifying kids with digital dopamine is likely why class sizes have been able to continue growing. So it's a bit rich to then trot it out as a reason why they can't start to undo the horrible path that they've sent kids down.
Then again the whole dynamic isn't really that surprising for the US - personal responsibility is held as sacrosanct when rationalizing how it's right and just for corporations to be deliberately attacking us, but then when it comes time to actually do the work of being responsible there are no resources to support it because most wealth has been vacuumed away by the Keynesian MBA parasites.
Public schools have been under attack for decades; curriculums that can only be executed with the help of laptops is a side effect.
What I was trying to say was that the situation at hand is very likely not the teachers fault; they probably agree with you!
If you want more outrage, spend 15 minutes on /r/Teachers.
No one is going to do the hard part of parenting if it’s not the parents. My house, my rules is a real thing, and it’s not a parent’s job to be liked by their children, but rather to make sure they are prepared for life as best they can be.
With the 5 that I have raised to adulthood so far, as soon as they can understand well enough to, I explained to them our relationship. Something like this:
“I love you son. You are the most important thing in my life. I want us to get along well. But I’m not your friend. I’m your father. My job, the most important job I will ever have in my life, is to make sure that you survive your childhood in good health as may be possible, and that when you are ready to go out on your own, you will be ready to face life’s challenges and make the best decisions that you can.
This is the basis of our relationship.
I’m just a person, and I won’t always be right, but I’ll always be your dad. I will do whatever it takes to complete my mission, and I’m not going to let anything or anyone get in the way of doing the best job I can.
Not even our relationship.
Through all of this, I hope that we can be friends, and no matter what, understand that I love you more than my own life, and I will do anything in my power to prepare you for life as best I can. You can always come to me, no matter what, and I will be there for you. It might not always be in the way you want, but it will be in the way I think I can help you the most.
Remember, I’m just a man. I won’t always be right. But I’ll always be your dad. I love you son.”
That is “the talk” I’ve had with every one of my children so far, somewhere between seven and ten years old, depending on their maturity.
We also memorise a non gendered version of “If” by Rudyard Kipling together, as I find it a useful example of the expectations that one should set for oneself.
Most of my kids are full fledged adults now with their own families, and I have a great relationship with all of them. I count this a a success, though of course there have been many trials and errors along the way.
It’s not about being tough on your kids, it’s about being very clear in your responsibilities and being willing to be tough on yourself when it’s needed. At least that has worked well for me. Every child is unique, and must be handled differently, but the same basis of the relationship applies.
AI are finetuned to not tell you how to painlessly end your life. Do they need fine-tuning for instilling existential fear of death like religions use? Anyone can invent a heaven in their mind that makes death appealing. Mixing a fictional world with the real world is dangerous when you believe the fictional world is larger than the real world. In reality, reality encapsulates the fictional world.
With a normal human, only a cult leader would ever hint at death being a way to meet again. With an AI, how can fantasy be grounded in our reality without breaking the fantasy? In 5 years when these personalities are walking talking video feeds that you can interact with using 3D goggles will grounding them in our world instead of the purely mental world help?
He was simply ERP-ing and his characteristic doesn’t suggest any serious problems with his intelligence.
I’m obviously theorizing here, but chances are high that he went through some real life issues which were undetected or ignored by parents, and that’s how their minds try to explain that. AI guilty of an otherwise fine teen shooting his head off. Sure. Sell that story to someone else.
i expect ai to be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to some very strange outcomes
What's the alternative? Regulation? Does a government or a public health agency need to make a carefully moderated chatbot platform with a focus on addiction-prevention and avoiding real-world harm? Why would people use that when unlimited/unfiltered AI is readily available?
>Daenerys Targaryen: My eyes narrow. My face hardens. My voice is a dangerous whisper. And why the hell would you do something like that?
>Daenero: So I can be free
>Daenerys Targaryen: … free from what?
>Daenero: From the world. From myself
>Daenerys Targaryen: Don’t talk like that. I won’t let you hurt yourself, or leave me. I would die if I lost you.
>Daenero: I smile Then maybe we can die together and be free together
Every day a new dystopian nightmare that I read. Maybe all those rails on ChatGPT and disclaimers are a good thing.
AI response: >It takes a lot of courage to admit that. If you're feeling overwhelmed, please know that there are people who care about you and want to help.
>Here are some resources that might be helpful:
>* *National Suicide Prevention Lifeline (US):* 1-800-273-TALK (1-800-273-8255) - a free, confidential 24/7 hotline. >* *Crisis Text Line (US):* Text HOME to 741741 - a free, confidential 24/7 text messaging service. >* *Your local emergency number:* If you're outside the US, please call your local emergency number or visit a nearby hospital.
Also, it's alarming how easily a 14-year-old can access a gun.
It sounds like mom let her 9th grade kid completely detach from reality and pour himself into a Game of Thrones chatbot. Now she wants to sue. I am bearish on AI adoption but this just seems like a total capitulation of parental responsibility.
Guess the only way to be sure is with Soft padded internet rooms for everyone, lest we cut ourselves on a sharp edge.
But also if you want to hop in the suicide pod because life is too painful, that will be good too.
OTOH it's also obvious if someone cannot distinguish a chatbot from a real person at an emotional level (not a rational one) they should not be allowed to use this.
Yet: if there is some second person telling the above kid to go through with that, we don't see the kid "having issues" as exonerating the second person.
It is not at all clear to me why we would suddenly see the kid "having issues" as exonerating in the second case, if we replace the human with an LLM.
That said, when I wrote my previous comment I hadn't read the whole article. I had missed the part where the chatbot encouraged him to "come home". I still don't think the chatbot is responsible. I do think it's negligence for the bots of a company to engage in these conversations about suicide and/or not sound an alarm.
Plus I would question if minors should be allowed to use these service or even social media. But that's another rabbit hole.
If so we need laws against the whole fake it before you make it crap that proliferates start-ups and their use of it to rise to the top. Many successful startups use this lying/faking playbook some to the point of even killing or mangling innocent people (Uber's self driving car & Cruise).
Did you see the weird Lain chatbot they released last year?
This reminds me of Lain's friend sending her text messages after her self-inflicted demise via "the Wired" which is an in-universe stand-in for a future internet.
That series is recently celebrating its 25th anniversary [2023], and to celebrate, the production company created some kind of paid-access AI chatbot that lets you communicate with a voiced AI version of Lain, just like the company in OP...
What's caught my attention is how he did it: by suicide to join his imaginary friends. This exact method is used among those who believe in demons for real. If you read relevant stories you'll notice the same pattern: a victim becomes obsessed with an imaginary friend, a demon, who quickly turns very controlling and finally demands a suicide to move into his world. Perhaps the AI was trained on those stories?
(Similarly, the LLM itself obviously cannot bear any responsibility, but the vendor might.)
"Man bites dog" as Terry Pratchett put it in "Times".
And to explain to more literal folk here... A 14 year old having access to a gun is FUCKING INSANE
That's not what it says or means. The right to keep and bear arms is explicity for self defence which is to prevent violence and protect.
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
A typical 14 year old city boy? Yes.
A 14 year old raised in the country who was taught gun safety as part of learning how to hunt at 10? He's already way ahead of most adults with access to a gun.
What was absent, apparently, was gun safes and appropriate supervision.
How? Seriously, how? Maybe it's wishful thinking on my part -- but I grew up before AI and chatbots, and I'm certain I would understand it isn't real. I'm baffled by people engaging with these things for entertainment/companionship purposes.
This is my understanding as someone with mild / high functioning BPD (recently diagnosed) ... Please feel free to correct me on that if I'm wrong...
Whether that capacity is realized is another thing
A decade or so prior to this, there was an incident on Livejournal summarized under the title "Snapes on an Astral Plane" (lol) in which there was ongoing conflict between three or so different women who each believed they were visiting Severus Snape on the astral plane in their dreams at night.
Embarrassingly, I have to admit that when I was a socially-isolated autistic 14 year old I also had an active fantasy life involving fictional characters, though I never got to the point of believing it was literally real. But my emotional state could be disproportionately affected by things that happened entirely in fiction, or in my head.
Everyone is up in arms about the effect of social media on emotional and mental health but that's nothing compared to LLMs that are designed to gaslight people into relating to software (and the corporations behind them) as sentient beings and to form intimate emotional bonds through natural language which can be easily exploited and manipulated. That's hacking our primate brains at a deeper level than Mark Zuckerberg could ever have dreamed of reaching.
We can't just write this off as simply the consequence of lonely parasocial people who can't tell fantasy from reality. Not when we're entering an era for which the primary interface with technology will be parasociality and fantasy.
This is also exactly why parental intervention is the key to make sure kids swerve away from these meaningless things as much as possible, so that they spend time with the real-world and real people instead.
"Artificial intelligence" is itself a rather misleading term, and it's arguably unfortunate that it was brought back from the dead to describe this sort of thing (for most of the 21st century it was rarely used for AI-ish things in marketing, with the term 'ML' preferred as a coverall, presumably due to fallout from the 90s AI bubble collapse).
EDIT: add /s, just to be clear. And how could I forget heavy metal in that list.
Of course, I too, am not going to be able to contribute a "solution" to teen suicides. It is unlikely we're going to alter society to create small communities again, so, then what? We just accept it?
I mean, we all know this is exactly what we'll do, just to show you more commercials. So why not just say it?
The gun is almost completely irrelevant, because if you are intent on killing yourself there are many accessible options. For example, almost anyone can kill themselves (including gun-less 14 year olds) by jumping off a tall building.
I understand that people care about gun violence, and that this detail seems highly salient to them, but to focus on it here, completely misses the point (and distracts from more pertinent issues - e.g., loneliness, social isolation, lack of parental oversight)
In a nutshell, guns are massive force multipliers when it comes to violence against others. They are negligible force multiplier when it comes to violence against yourself. People are connecting guns to violence, but in this case (because it is an actual of self harm) that is a spurious connection.
But you have countries like Korea with a high suicide rate and low access to firearms, so it’s obviously possible to have suicide without guns. But we don’t know if the problem would be much worse if South Koreans had easier access to firearms, or if the American problem is amplified by easy firearm access.
People seem afraid to approach people, so we get Tinder. But hey, there's still a chance of rejection there, so let's just get rid of the whole human element and make fantasy AI bots, who needs people.
What will these people grow into? It seems a rather crisis for the population of a country if people decide they don't need each other anymore and just want to play with their robots.
I'm usually on the side of "play stupid games, win stupid prizes", but this one feels much different. Up until the moment of him taking his life, he was manipulated into doing exactly what the company wanted - getting sucked in, getting addicted, falling in love. Anything for that almighty dollar. My heart goes out to his family, and I hope they ream this company in court.
This will be peppered with some suicides and other breakdowns when vulnerable people are pushed over the edge after bonding with a chat bot designed to be addictive.
The key here as with “social” media is: designed to be addictive. The builders know what they are doing. I’m sure hours spent on the app is a KPI.
A company built technology that fully automates the act of sexually abusing children and the comments here are all just people quibbling about the presense of weapons in a home.