With AI chatbots, Big Tech is moving fast and breaking people
arstechnica.com
arstechnica.com
But it goes to show just how vulnerable a lot of humans are.
By all accounts, GPT-4o isn't malicious. It has no long term plans - it just wants the user to like it. It still does this kind of thing to people. An actual malicious AI capable of long term planning would be able to do that, and worse.
One of the most dangerous and exploitable systems you can access online is a human.
OpenAI thinks that as long as you aren't using dating terminology or asking to build bombs, you're probably safe.
- What does having an unquestioning mirror that is always gracious and reflects your own biases back at you, do to the mind in the long term? We don't know.
- How do you know, that after 2-3 years of this use, without any particular warning, it will not suddenly escalate into a delusion spiral? We don't know.
- Does this affect childhood development to have no pushback on their personal feelings and opinions? We don't know.
- Does this affect people's very willingness to learn, and to endure intellectual struggle? We don't know.
- Does this affect long-term retention, as losing even an imperceptible 2% of your knowledge every month, is 48% lost after 2 years? We don't know.
- How does this affect a population of which 27% have experienced a mental health issue in the last year, and of which 9% are experiencing depression? We don't know.
- Are all these problems solvable with any amount of education or warning labels, given that human nature has not changed and will not change? We don't know.
- Can this be prevented with any amount of AI guardrails or ethics training, or is it inherent to the structure of the system itself? We don't know.
- Will this eventually be considered so dangerous, that we need black-box warning labels similar to cigarettes and possibly social media? We don't know.
- Will this eventually be considered so insolvably risky for human cognition, that we outright restrain use without a license, either like driving at best, or asbestos at worst? We don't know.
- Will any employer be able to obtain insurance for AI use, if there is even a 0.2% documented chance that any employee could go psychotic from using it, completely at random? We don't know.
- Will said employees, if they do go psychotic from a work-mandated AI chatbot, successfully sue their employers for damages? We don't know.
Education I guess is the answer, but we don't even seem to be able to properly educate children about pornography and social media, let alone the subtleties of interacting with AIs..
If these are real problems, hell no, because education has never forced anyone to do anything right.
You can yell about the dangers of STDs until you are blue in the face, but 20% of the population carries one.
You can yell about the dangers of speeding just 5 MPH with graphic imagery in drivers ed, but 35% of adults go 10+ MPH over in residential areas every month.
The answer, quite literally, is possibly age verification, outright bans, or mandatory licensing. None of which are priced into the stock market.
However, any government will quickly realize that having up to a quarter of your population vulnerable to developing psychosis and mental problems (due to 27% of the population experiencing at least some form of mental illness just in the last year, making them presumably more vulnerable to AI) is absolutely literally financially impossible to allow freedom in experimenting with.
So what then, ban sex? Good luck.
Education is really the only answer here. There are clearly challenges in doing it right but fucked if I don't think it'd be worse not to even try...
It can't. Full stop. It needs to be a choice of the people involved, to do the hard good thing, unless it is otherwise imposed on them. We enjoy "freedom" in the sense that the good choice is often not imposed, because enough people make the good decision regardless. If AI successfully convinces enough people to not make that choice, despite all the lecturing on why you should make the proper choice, we're screwed unless we clamp down hard.
I do know that a society with the level of cognitive dependency that AI risks introducing, and is already introducing, literally cannot function without collapse. Do you want a society intellectually dependent on advanced manufacturing techniques in one of the most geopolitically at-risk regions of the world?
We _kind_ of do: https://en.wikipedia.org/wiki/Folie_à_deux
Though I suppose this is more Folie à Un.
That's just what a malicious AI would want you to think...
But seriously, there doesn't need to be any "intent" for something to be harmful. A deadly virus has no will or agency, and particular individuals may be completely immune to it. But released on a population the results are catastrophic. We have to accept that perhaps for the first time ever we've created something we can't fully comprehend or control, and that regardless of our best wishes it will be harmful in ways we could never have imagined. If it were as simple as prompting the next model against this stuff, it would have already been done.
It uses words and phrases most commonly associated with other text that humans have labeled as "helpful", "friendly", etc. That sort of thing.
That's different than wanting. It's text probabilities in a box. It can't want anything.
I am normally not a stickler, but in this case the distinction matters.
We already know that you can train an "evil" AI by fine-tuning a "normal" AI on sequences of "evil" numbers. How the fuck does that even work? It works because the fine-tuning process shifts the AI towards "the kind of AI that would constantly generate evil numbers when asked to generate any numbers".
And what kind of AI would do that? The evil kind.
AI "wants", "preferences" and even "personality traits" are no less real than configuration files or build scripts. Except we have no way of viewing or editing them directly - but we know that they can be adjusted during AI training, sometimes in unwanted ways.
An AI that was fried with RL on user preference data? It, in a very real way, wants to flatter and affirm the user at every opportunity.
OpenAI attempted to nuke their overly-agreeable LLM (GPT-4o). After user outcry they brought it back. I wouldn't be too sure that companies will do the responsible thing here; there's a pretty ugly history of companies who sell to addicts deliberately making their products more addictive and appealing to said addicts (the tobacco industry is particularly notorious for this).
Like, one view would be that LLM 'companions' and 'therapists' are inherently harmful, or at least so likely to be net-harmful that making them available would be unjustifiable. But that's certainly not the view that the vendors are taking.
> It has no long term plans - it just wants the user to like it.
To be clear, it doesn't 'want' anything.
Some people can't help themselves and don't use these tools appropriately.
"but it's getting cheaper and cheaper to run inference" they say. To which I say:
ok bud sure thing, this isn't a technology governed by Moore. We'll see.
But that's not what I'm talking about, I'm talking specifically about people who've made a SoTA model their buddy, like the people who were sad when 4o disappeared. Users of character.ai. That sort of thing. It's going to get very, very expensive and provides very little value. People are struggling with rent. These services won't be able to survive, I hope, purely through causing psychosis in vulnerable people.
you haven't met the Whales (big spenders) in the loneliness-epidemic industry (e.g. OnlyFans and the like)
It's not like the "real women" of OnlyFans are consistently real, or women. And there's some percentage that are already AI-by-proxy. There's definitely opportunity for someone to just skip the middleman.
But that isn't what I've seen done when people said they did that. Instead they just told ChatGPT a bit about the person and asked it to playact. The result was nothing like the person-- just the same pathetic ChatGPT persona, but in their confusion, grief, and vulnerability they thought it was a recreation of the deceased person.
A particularly shocking and public example is the Jim Acosta interview of the simulacra of a parkland shooting victim.
I've found Claude to be an excellent tool to facilitate introspective psychoanalysis. Unlike most human therapists I've worked with, Claude will call me on my shit and won't be talked into agreeing with my neurotic fantasies (if prompted correctly).
It's called unconscious intention, and here's a pretty interesting paper that'll bring you up to speed: https://irl.umsl.edu/cgi/viewcontent.cgi?article=1206&contex...
In your provided example, the user is obviously not trying to manipulate someone into questioning their sanity, nor power of reasoning. Quite the opposite. Lying to themselves (your example) for sure.
https://philpapers.org/rec/MCGAIG
https://www.psychologytoday.com/us/blog/emotional-sobriety/2...
It can drive people further and further into their own personal delusions or mental health problems.
You may think it's being critical of you, but it's not. It's ultimately interacting you on your terms, saying what you want to hear when you want to hear it. That's not how therapy works.
That's been my experience with human therapists. When I tell Claude to stop being sycophantic, it complies. When I tell a human to stop being sycophantic, they get defensive.
I agree that an ideal human therapists would be better than Claude, but most that I've worked with are far from ideal. Most are not very bright, easily manipulated, and quick to defensiveness when questioned. And Claude won't try to get me to take random meds with the only justification for the specific medication being 'got to start somewhere'.
To be fair these are probably the same people who would have been having these conversations with squirrels in the park otherwise.
I suppose the only hope here is that its economic value comes mainly from its productive output (summarization, code, writing, text transformation whathaveyou)
And Snapchat, TikTok, Facebook, YouTube, Instagram, more?.. are just it. They waste time worth $50-$100 just to make $0.05 in ads.
LLMs seem to be far from that.
* I might be ignorant of the real picture, please correct me if I'm wrong and not aware of really evil LLM companies.
So it's platform partners plus filtered selection putting the tuned dopamine poison in front of people for the moment, but it's absolutely happening. And eventually the platform owners can and will cut out the middlemen.
Though, interestingly, it has been _predicted_ for a while. It's a bit blink-and-you'll-miss-it, but the Ministry of Truth in 1984 has book-writing machines!
For those of you who, thankfully, don't have personal experience, it generally goes like this: reasonable-ish individual starts using AI and, in turn, their AI either develops or is prompt-instructed to have certain personality traits. LLMs are pretty good at this.
Once the "personality" develops, the model reinforces ideas that the user puts forth. These can range from emotional (such as the subreddit /r/MyBoyfriendIsAI) to scientific conspiracies ("yes, you've made a groundbreaking discovery!").
It's easy to shrug these instances off as unimportant or rare, but I've personally witnessed a handful of people diving off the deep-end, so to speak. Safety is important, and it's something many companies are failing to adequately address.
It'll be interesting to see where this leads over the next year or so, as the technology -- or at least quality of models -- continues to improve.
First, hallucinations aren't even the problem we're talking about. Second, there have been marked improvements on hallucination rates with just about every frontier release (o3 being the notable outlier).
It might be an actual order-of-magnitude increase from the background rates. It might be basically the same headcount, but with psychosis expressed in a new notable way. I.e. that guy who would be planning to build a perpetual motion machine to disprove quantum physics is now doing the same, but with an AI chatbot at his side - helping him put his ramblings into a marginally more readable form.
How does one "almost attempt" something?
Mostly a joke but one the common properties in people with an delusional or otherwise unhealthy pattern of AI use appears to be substituting it for all their own choices, even where it's obviously inappropriate. E.g. if you try to intervene in their conduct, you'll just get an AI generated reply.
(It's a fairly awkward phrase, and you wouldn't say "I almost attempted to run a marathon", say, but "attempt suicide" has a fairly specific commonly-understood meaning where 'attempt' is unusually restrictive vs general usage)
But in areas I am not, I am still skeptical, but enthusiastic — that they’ll be vindicated maybe in a decade. Any physics or QM experts can opine on this? https://magarshak.com/blog/?p=568