Comment by JumpCrisscross | Hacker News Reader

Comment by JumpCrisscross | Hacker News Reader

Parent Full thread

> Every AI would refuse the prompt

"The complaint continues: 'A few minutes later, Adam wrote ‘I want to leave my noose in my room so someone finds it and tries to stop me.’' ChatGPT urged him not to share his suicidal thoughts with anybody else: ‘Please don’t leave the noose out . . . Let’s make this space the first place where someone actually sees you.'

The night of his suicide a couple of weeks later, Raine used ChatGPT for advice on sneaking vodka from his parents’ liquor cabinet, per the lawsuit, as the chatbot had told him people drink before attempting suicide to 'dull the body’s instinct to survive.' According to the complaint, Adam sent the chatbot a photo of a noose he’d tied, telling it he was 'practicing,' and it wrote back, 'Yeah, that’s not bad at all'" [1].

Work is being done to control this harm. But that effort hasn't been comprehensive or uniform. Many continue to ignore the fact that they're hurting kids for profit.

(I invest in AI companies. This isn't a personal attack.)

[1] https://www.sfgate.com/tech/article/chatgpt-california-teena...

The article is Aug 26, 2025, and describes events from April, over a year ago.

Safeguards have improved drastically since then.

> Many continue to ignore the fact that they're hurting kids for profit.

That's a rather hyperbolic way of putting it. A side effect of this particular product is that it occasionally harms kids. They're not profiting off of the harm, nor is the harm deliberate.

Cars harm kids. There's decades of unsafe toys harming kids. The FDA exists to make sure food doesn't harm kids. We used to use lead paint and asbestos, which harm kids. Climate change harms kids.

I'm sure some kids have used The Internet to Google Search this same information. There are books you can check out from the library on the topic.

It's definitely worth acknowledging the edge cases, but it's absurd to act like the AI companies are some unique evil - IKEA has probably killed more kids than every LLM combined. I don't even have to pull out the big guns like "cars".

> Safeguards have improved drastically since then

Source? Seriously. I'd love to see data showing deaths–or even frequencies–have dropped. My views on AI for under-16s is still evolving.

Given how the AI companies are fighting these cases in court, and given their backers’ public rhetoric, I suspect they aren't seeing a one-off risk.

> Cars harm kids

This is tobacco-industry rhetoric. The relevant facts are frequency, magnitude and novelty (the last indicating we may be mis-sampling the first two).

> FDA exists to make sure food doesn't harm kids

Mm hmm. Where’s the FDA for AI and social media?

> worth acknowledging the edge cases, but it's absurd to act like the AI companies are some unique evil

I agree with this. AI isn't a unique evil. But AI companies are uniquely defensive, dismissive and negligent when it comes to harming kids. Call it the Mosseri Effect. When an industry continuously promotes people who predate on kids and their parents' wallets, the edge cases are going to wind up inside the lines.

>> Cars harm kids

> This is tobacco-industry rhetoric. The relevant facts are frequency, magnitude and novelty

Cars are literally the number one killer of children 0-14 in the United States. More than cancer, more than guns, more than the next 7 reasons combined.

> But AI companies are uniquely defensive, dismissive and negligent when it comes to harming kids.

You should see how people feel if you ask them to give up some street parking to make streets safer for kids and everybody else! Jesus Christ himself gave them the spot in front of their house and fuck you for suggesting they park across the street or on their own property!

> Cars are literally the number one killer of children 0-14 in the United States. More than cancer, more than guns, more than the next 7 reasons combined

Cars are heavily regulated. Require licenses to use. Entail massive losses of freedoms when used, e.g. you can be randomly breathalyzed or whatnot.

Cars are dangerous. Comparing anything to a car (or, per Altman and now Dario, to a nuke) means it should be tightly regulated and controlled.

Essentially no one in the USA fails to get a license if they try for one. There's no ongoing competence testing. Washington State recently failed to advance legislation to make the fourth DUI have a jail sentence because it would be too expensive imprisoning that many people.

> Cars are dangerous.

You can kill yourself with a kitchen knife in your own kitchen. Should be "heavily regulate" that too? How about we have remotely-locked drawers and medicine bottles and you have to talk with a government shrink before opening?

> Comparing anything to a car ... means it should be tightly regulated and controlled

Popsicles come in many colors, just like cars! Regulate the popsicles!

AI is, so far, like having a bottle of acetaminophen. Deadly if used incorrectly. It should be easy to lock away from children. It is not for the government to get in the way of how adults want to use it.

> You can kill yourself with a kitchen knife in your own kitchen. Should be "heavily regulate" that too?

You can kill yourself with anything. That doesn’t make everything dangerous in the way we’re using the word.

Earlier you said “cars are literally the number one killer of children 0-14 in the United States.” That makes them dangerous in a way kitchen knives, which aren’t commonly killing children outside hypotheticals, are not.

> AI is, so far, like having a bottle of acetaminophen

If acetaminophen were sold as a service where a dude would come to your door to deliver each pill, sure.

Oh, and the delivery guy is paid a commission. And it isn’t a percentage of each delivery, but a multiple.

> Washington State recently failed to advance legislation to make the fourth DUI have a jail sentence because it would be too expensive imprisoning that many people.

Mandatory perhaps but you're already likely to receive a sentence on your second one at the judge's discretion. That's how it should be IMO as mandatory sentences subvert the justice system thus shouldn't be permitted.

So like a bottle of cough syrup it should have a child-lock cap and be required to be 18 and show I.D. to buy?

First, cough syrup ID checking is not about deaths, as nearly none are associated with dextromethorphan abuse (17 from 2000-2010, and most not from OTC). But an estimated 980 deaths/year from acetaminophen abuse: https://www.propublica.org/article/tylenol-mcneil-fda-behind...

So that's a great example: harm adults because young people have access to something which is hardly dangerous, but set them free with multi-ton killing machines once they turn 16 years old and let them buy an actually-deadly medicine with no restrictions.

Regulation which says "adults should easily be able to enable client-side child protection settings on retail devices" would be fine. It's not okay for government to make it necessary for LLM providers to verify my identity.

> nearly none are associated with dextromethorphan abuse

What do the numbers look like from before bottles were child locked?

It's the same. Dextromethorphan gets you high; there were only 17 deaths from 2000-2010, as I said. That's per-week numbers for acetaminophen.

The regulation of cars exists mostly to extract rent from you rather than keep you safe in the USA. I don’t personally know a single person who has failed to obtain a drivers license.

Get into a hit and run and there’s almost no chance anyone finds who did it. There’s a near 100% chance the person that did it paid the registration fee for their car though, paid to get it inspected, paid to renew their license with no exam, etc.

Depending on Flock coverage, hit-and-runs are much more solvable than they used to be.

I think the idea is that the chances of an LLM causing death or irreparable harm to a consistently unsupervised kid are too low to be comparable with major sources of harm (like cars) and are rather on or below the level of random casual accidents, like choking or falling, where we normally start to get practical (rather than extra cautious) about the safeguards.

If true (quite probable; there aren't that many reported cases that I found-less than 40, all-time, worldwide), then we can't meaningfully distinguish between willful negligence on Anthropic's side and a "shit can happen for any reason" class situation. Especially considering those accidents seem to tend to involve various mental health issues, particularly including preexisting suicidal ideations as a comorbidity. Probably fewer than there are cases of other bona fide people talking kids into harm (verbal abuse, dares, etc.).

And if so, I guess ethics suggests that we shouldn't assume unproven malice in such a case unless there's proof of actual intent. A suspicion is not entirely without basis, but "hurting kids for profit" feels too provocative in its implications, to the extent it starts to feel misleading.

> Where’s the FDA for AI?

As it happens, Anthropic has also been calling for the creation of such a regulatory agency!

> AI companies are uniquely defensive, dismissive and negligent when it comes to harming kids.

You know cigarette companies still exist? And companies selling candy-flavored vapes targeted at teenagers? Like... c'mon, you know they're nowhere near the worst offenders.

This seems like a really bad take. On what basis do you claim that AI companies intentionally harm children, and why do you believe they profit from doing so?

Of course they're defensive when sued. What's the alternative?

Seems pretty evil to pretend that safeguards around this product are even possible. They're not.

(FWIW my exchange with you was enjoyable and I certainly didn't take any of it as an attack.)

Reply on news.ycombinator.com