Hacker News Reader

There is something deeply disturbing to me about these "conversations".

Reminds me of a Philip K. Dick short story titled Progeny. In its universe children are raised exclusively by robots. Unlike humans, they never make mistakes or commit abuse. The child, once grown, ends up seeing his Father as an animal and the robots as his kindred. In the last pages, he chooses the sterile world of the robots instead of joining his Dad's work/explorations in the far reaches of the solar system.

Our current chatbots are still flawed, but they're still sterile in the sense that you can trash them and start anew at any moment. You're never forced to converse with someone who is uninteresting, or even annoying. Yet, these are the very things that grow people.

Yeah, good luck to these kids in forming relationships with the roughly 100% of human beings (aside from paid therapists) who really have no interest in hearing your anguish non-stop.

It's probably a good thing most of us force our kids to spend a minimum of 7 hours/day, 200 days/year surrounded by a couple hundred similarly-aged kids and a few dozen staff members, with an unusually high variance of personalities (compared to adult life).

It strikes me as something that can be incredibly useful or do great harm, depending on dosage. A selection of conversation partners at your fingertips, and you can freely test reactions without risking harm to a relationship. At worst you reset it. Maybe you can even just roll back the last couple messages and try a different angle. Sounds like a great way to enhance social skills. Yet as you point out, healthy development also requires that you deal with actual humans, with all the stakes and issues that come with that.

>without risking harm to a relationship

Only from a very narrow perspective. Opening yourself up and being real with people is how relationships form. If you test every conversation you are going to have with someone before having it, then the 3rd party basically has a relationship with an AI, not with you.

Now testing every conversation is extreme, but there is harm any time a human reaches out to a computer for social interaction instead of other humans.

That "instead of other humans" part is doing a lot of heavy lifting here. What if it's "instead of total isolation" or "instead of parasocial interactions" or "instead of exploitative interactions"? There are many cases that are worse than a person chatting with a robot.

People who are used to working with an undo stack (or with savegame states) are usually terrified when they suddenly have to make do in an environment where mistakes have consequences. They (we) either freeze or go full nihilistic, completely incapable of finding a productive balance between diligence and risk-taking.

If by social skills you mean high performance manipulators, yes you would get some of those. But for everybody else, it would be a substitute to social interaction, not a preparation for.

Or we discover roguelikes!

It's very rare that you would ever say something that would have real damage that couldn't be resolved by a genuine apology. Having to actually go through an awkward moment and resolving it is a real skill that shouldn't be substituted with deleting the chatbot and spawning in a new one.

> "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point

And the verses he found objectionable are really there. Are they also suing the Gideons for not ripping those pages out of their hotel room Bibles (or maybe they think you should have to prove you're over 18 before reading it)?

Keep kids away! Too much weird incest stuff in that book.

I think the suggestion of violence is actually on page 31 (paragraph 103), though it's not a directive.

It does seem a bit wild to me that companies are betting their existence on relatively unpredictable algorithms, and I don't think they should be given any 'benefit of the doubt'.

No no, they're betting other people's children on relatively unpredictable algorithms. Totally different!

We mean the same page. The one that has a 28 written on it but is the 31st in the pdf. I didn't notice the discrepancy.

Given the technology we have, I'm not entirely sure what Character AI could have done differently here. Granted, they could build in more safeguards, and adjust the models a bit. But their entire selling point are chat bots that play a pre-agreed persona. A too sanitized version that constantly breaks character would ruin that. And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied.

>"And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied."

I agree with everything you're saying, but there are no legal protections for incitements to violence, or other problematic communications (such as libel) by an LLM. It may be that they provide a very valuable service (though I don't see that), but the risk of them crafting problematic messages is too high to be economically viable (which is how it seems to me).

As it stands, this LLM seems analogous to a low-cost, remote children’s entertainer which acts as a foolish enabler of children’s impulses.

The cynic would say that if their business model isn't viable in the legal framework that's just because they didn't scale fast enough. After all Uber and AirBnB have gotten away with a lot of illegal stuff.

But yes, maybe a service such as this can't exist in our legal framework. Which on the internet likely just means that someone will launch a more shady version in a more favorable jurisdiction. Of course that shouldn't preclude us from shutting down this version if it turns out to be too harmful. But if the demand is there, finding a legal pathway to a responsibly managed version would be preferable (not that this one is perfectly managed by any means)

There has to be a 'reasonable person' factor here - otherwise if I'm watching Henry V I can sue everyone and his mother because the actor 'directed me to take up arms'! I never wanted to go into the breach, damn you Henry.

> I'm not entirely sure what Character AI could have done differently here.

Not offer the product? Stop the chat when it goes off the rails?

> I'm not entirely sure what Character AI could have done differently here.

You're taking it as a given that Character AI should exist. It is not a person, but an offering of a company made up of people. Its founders could have started a different business altogether, for example. Not all ideas are worth persuing, and some are downright harmful.

Page 5 is pretty strong too. And that's as far as I've gotten.

And paragraph 66, page 18 is super creepy. The various posters apparently defending this machine are disturbing. Maybe some adults wish that as a kid they'd had a secret friend to tell them how full of shit their parents - and wouldn't have position if that friend was either real or imagined by them. But synthesized algorithms that clearly are emulating the behavior of villains from thrillers should be avoided, woah...

I think it’s more that some people are excited by the prospects of further progress in this area, and are afraid that cases like this will stunt the progress (if successful).

Well, the founders already won, according to the article:

> Google does not own Character.AI, but it reportedly invested nearly $3 billion to re-hire Character.AI's founders, former Google researchers Noam Shazeer and Daniel De Freitas, and to license Character.AI technology.

There's a reason why licensed therapists are the ones giving diagnoses of "abuse", etc. The problem with these AIs is that they use adult-sounding language, to be an echo chamber to children - thus appearing like a voice of authority, or someone with more knowledge (including citing media articles the child may not have even been aware of) when in fact they're just parroting the child back.

I don't know if there is actual abuse or not, but the way Character.AI presents themselves in these conversations toes a very slimy grey line, in my opinion. If you go to their site and search "Therapist", you'll find 30+ bots claiming to be Therapists, including "Dirty Therapist" that will give you therapy in a "bad, naughty way."

I really want to emphasize the above post is filled with lies. The "incite to kill" part is the first image in both the complaint and the article and it's fairly unambiguous. The image on page 28 is, creepily enough, the bot making "sexual advances" at the kid.

I find people defending and lying about this sort of thing disturbing, as I think many would. WTF is wrong with hn posters lately.

In the first screen shot I see, the bot:

1. Makes not using a phone a huge issue, implying it's a kind of abuse...

2. Indeed hints at killing: "I'm not surprised when I read the news and see ... "child kills parents after a decade of physical and emotional abuse"...

I mean, saying "empathy" is statements akin to OMG, no phone time, that's horrific, seems quite inaccurate.

Just remember that you are seeing one side of this story. The mother may well be one of the best parents but has a bad kid. We have no idea. (most likely mother is not perfect, but no other parents are)

Edit: we see both sides through very limited information since we only get what is in the legal filing.

We're seeing the mother's side of the story. That's the one side we're seeing.

My reading is pretty much the same as yours. I think of it in terms of tuples:

   { parents, child, AI_character, lawsuit_dollars }

The AI was trained to minimize lawsuit_dollars. The first two were selected to maximize it. Selected as in "drawn from a pool," not that they necessarily made anything up.

It's obvious that parents and/or child can manipulate the character in the direction of a judgment > 0. It'd be nice if the legal system made sure it's not what happened here.

> The AI was trained to minimize lawsuit_dollars.

That seems wrong. The null AI would have been better at minimizing legal liability. The actual character.ai to some extent prioritized user engagement over a fear of lawsuits.

Probably it's more correct to say that the AI was chosen to maximize lawsuit_dollars. The parents and child could have conspired to make the AI more like Barney, and no one would have entertained a lawsuit.

OK, it seems like a nitpick argument, but I'll refine my statement, even if doing so obfuscates it and does not change the conclusion.

The AI was trained to maximize profit, defined as net profit before lawsuits (NPBL) minus lawsuits. Obviously the null AI has a NPBL of zero, so it's eliminated from the start. We can expect NPBL to be primarily a function of userbase minus training costs. Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict, so the loss function can target both. It seems to me that the additional training costs to minimize lawsuits (that is, holding userbase constant) pay off handsomely in terms of reduced liability. Therefore, the resulting AI is approximately the same as if it was trained primarily to minimize lawsuits.

> Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict

Not at all obvious (and I'd naively think it's wrong).

So you think it's more than "not much." How much exactly? A 10% increase in userbase at peak-lawsuit?

It's obviously a function of product design. If they made a celebrity fake nudes generator they might get more users. But within the confines of the product they're actually making, I doubt they could budge the userbase by more than a couple percent by risking more lawsuits.

My impression: In these early days of the technology there's huge uncertainty over what gets you users and also over what gets you lawsuits. People are trying to do all kinds of things with these models, most of which don't quite work at the moment. On the lawsuit side, there's potential copyright claims, there's things like the present article, there's celebrities suing you because they think the model is imitating them, there's someone suing you because the model insults or defames them (even if the model speaks the truth!), there's Elon suing you for the LOLs... As you're hoping to go global, there's potential lawsuits each jurisdiction which you don't even have the resources to fully evaluate the potential of.

You say that both factors are clear "within the confines of the product", but I'm not convinced there even are such clearcut "confines" of the product. To enter this market and survive, I'd think those confines woud have to be pretty flexible.

It's a 17 year old. Thinking back when I was 17 year old, I would've been very pissed as well if my parents took away my phone. And then especially if they went ahead and searched through it to find those messages. If they had friends I could see their teenage friends saying the exact same things as the AI did there.

Those screenshots and with the AI there, it does manage to make me not like the parents at all though. So AI maybe is quite convincing. If things go into that place where AI can do it, and parents blame the AI for it, when their kid is 17, it's almost like the AI was in the right there, that the parents were looking to just play victims. Blaming AI for being overly controlling and losing trust with their child.