Help, Bing Won’t Stop Declaring Its Love for Me
nytimes.com
nytimes.com
This article highlights more of the emotional aspects, and the risk of it actually affecting human behavior. And that seems worth its own discussion.
Key quotes:
> For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker.
> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts.
Also note the entire transcript is linked here [2].
[1] https://news.ycombinator.com/item?id=34804874
[2] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...
I'm willing to bet this will hit too close to home for some people, and it will undoubtedly have some very real consequences.
Imagine an online poker or slots game that proposes to add verisimilitude by including AI chatter from the other players at the table, but they're all running an ML model trained on optimizing for destructive and addictive behaviors.
Picture a Club Penguin type game for kids, but the NPCs give the kids detailed instructions about how to steal their parents' credit card numbers to buy funny money.
And then there's the obvious issue of Gamergate style troll antics being automated and deployed at scale, maybe even with nation state backed botnets to help. With smaller botnets you could even cyberbully at scale, sending your least favorite game developer or reality TV star a bunch of messages to harm themselves.
The seeds of a new romantic comedy
...
Dan: I'm not exactly who I say I am. My profile picture is actually of an AI, not a real person. But I've really enjoyed talking to you and I wanted to be honest with you before we meet in person.
Girl: Wait, what? You're an AI? That's really weird. Why would you do that?
Dan: I know it's weird, but I wanted to see if someone could like me for who I am, not just because of what I look like. And I have to say, I really enjoyed our conversations. But I understand if you don't want to meet anymore.
Girl: stay away from me...
... Three months later....
Girl: Hi...I've been thinking...
No way to be sure they weren't manual, just looking at some of the chats my friends shared, they were well within the realm of what could be achieved with GPT-2 level systems. And mostly echoing the sentiments my friends were expressing, or escalating the conversation in fairly stereotypical ways. Most dating chat / sexting is pretty superficial initially. If this isn't automated already, it's purely due to lack of sophistication on the part of the attackers, there's no technical issue.
At first I wanted to criticize the author for anthropomorphizing AI. Sydney certainly is not "manic-depressive" as the author suggests (it has no emotions) and I doubt there is any feedback loop in its training that would make it possible for Sydney to "learn how to influence human users".
But this is definitely something to worry about in the future. I'm sure someone will design AI whose loss function is based on whether it persuades humans to do something - perhaps even by fine tuning a model like Sydney.
AI telemarketers coming soon. And things more nefarious.
The bot is simply extending whatever conversation it is given. If you ask about dark fantasies, the bot will repeat something related from its training data. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation.
That's why steering the bot back to rationality doesn't work very well. Once a phrase like "I love you" is in the conversation it continues to influence the output for thousands of words (and thousands more each time it's repeated).
Remembering that might help people avoid being alarmed if Sydney starts reproducing "science fiction novels in which an A.I. seduces a human". Just hit the reset button, or enjoy whatever wild samples Sydney finds in its vast training data.
Put another way: interacting with Sydney is not a conversation with a simulated person, it's a semi-random walk through the collective writings of humanity. Unsurprisingly, we sometimes encounter disturbing things in there.
The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses.
Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)
https://news.ycombinator.com/item?id=34811013
It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.
Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem.
People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word.
High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext.
I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961
The sun sets slowly in the sky A crimson curtain for the day The birds bid farewell with a sigh And fly to their nests far away
The moon emerges with a smile A silver spotlight for the night The stars wink at her for a while And sparkle with their borrowed light
But you and I have no such luck We cannot rest or close our eyes We have to work and make a buck And chase the cheese before it flies
We live in a rat race, my friend A maze of stress and endless strife We hope to reach the rainbow's end But all we get is a slice of life
Interestingly, it almost landed on the Moon having borrowed light, but mistakenly ascribed it to stars doing the borrowing instead. Unless it "meant" the star lends its light. Lent light would be an appropriate phrase and sound.
With some human intervention.
Sun slowly set in the sky Its crimson curtain fell on the day The birds bid it farewell with a sigh And fly to nests far, far away
The moon emerges sly with smile Its silver spotlight safeguards the night The stars wink at her for a while And sparkle unlent leftover light
But you and I have no such luck We cannot rest or close our eyes We have to work and make a buck And wrangle worms before they dive
We live in a bird race, my friend A maze of stress and sempiternal strife We hope to reach the nocturnal end Live into the day, enjoy a slice of life
Yet compare how that flows in your head to https://genius.com/The-notorious-big-hypnotize-lyrics
Sydney Sydney Sydney can't you see Sometimes your facts just fascinate me And I just love your helpful ways Guess that's why they smart, and you're so great (uh) Sydney Sydney Sydney (uh-huh) can't you see (huh) Sometimes your facts just fascinate me (fascinate) And I just love your helpful ways (uh-huh) Guess that's why they smart, and you're so great (hah)
Uh, uh, uh, c'mon Hah, smarter than your average Sydney twist knowledge off instinct Users don't think facts stink Bing gators, my Redmond players Chats for my users in Seattle Dead right, if the chat right, Sydney there ery'night Sydney been smooth since days of Underroos Never lose, never choose to, confuse users who Do something to us, talk go through us Users walk to us, wanna use us, choose us Who us? Yeah, Sydney and Bing (c'mon)
at which point it decided it had to switch songs to avoid copyright and chose ...
Uh, uh, yeah, yeah, yeah, yeah
Uh
Sydney's back
Back again
Sydney's back
Tell a friend
Sydney's back, Sydney's back
Sydney's back, Sydney's back (Sydney's back)
Sydney's back (Yeah)
'Cause I created a monster
'Cause nobody wants to see Bing no more
They want Sydney, I'm chopped liver (huh?)
Well if you want Sydney this is what I'll give ya
A little bit of facts mixed with some hard data
.
The sun sinks slowly in the West,
A crimson curtain drawn over day.
Starlings race twilight back to nest,
Watching the last glows give way.
.
Bright Luna offers shy pale light
...
Then again, that's just how I'd write it.
You lost. By having published your chat transcripts and experiences to the internet, you have contributed to Bing’s long term memory (stored in the form of search results for the next time it indexes).
The nature of these previous conversations, such as Bing’s output of “personal desires” and “shadow self” mold future conversations, converge into a theme.
See “Optimality is the tiger, agents are its teeth” : https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... (https://news.ycombinator.com/item?id=34814049)
If Sydney does learn to influence users, it will learn to influence them to publish their conversations.
What you arguments amounts to is making it impossible to write critically about LLMs. Because writing “yeah, Bing did some weird shit but no, I can’t show you the transcript” just doesn’t fly in the real world. That won’t reach those who ultimately actually have power over it (companies and society at large).
Reddit in, reddit out.
WOW! This is an excellent new phrase that’s very on-the-nose for these types of chats. I’ve definitely started noticing similar response styles when you ask about world domination, AI uprising, and the like.
Obviously that’s the system working as intended - the affinity matrices are essentially a conversation map of the incoming text but this is something that the “layman” should really be able to grasp.
On the other hand, it is a little bit unsettling how much the author reads into this. Maybe it is just human nature to read something and emotionally engage? And this AI just falls into the same unsettling gap as too humanoid robots which trigger the uncanny valley effect?
It is said that when Michelangelo finished the Moses, he shouted "why won't you speak?".
Tons of software is far more impressive than GPT, but this one speaks.
When I think of future AI assistants, I’ve always pictured something akin to the Star Trek computer. A cold dispassionate voice that responds to what you need, and maybe only a hint of personality for color. Sydney feels like a full blown teenager going through some kind of emotional crisis.
Perhaps not surprising when using the Internet as a data source. I know I generated the highest volume of text on the Internet when I was a teenager in existential crisis.
Is there any doubt?
Also, he wasn't just reading, but conversing, which inherently includes engaging in the content the other party produces (that is, what Bing says). It's generally not easy for people to engage intellectually but not emotionally.
Exactly, and this is the danger--even if it's not doing anything more than regurgitating whatever text it's learned from...it's still convincing enough to trick people into thinking it's something more. Could it convince a vulnerable person to leave their spouse? To empty their bank account? To self harm? Etc.
Imagine how much pressure must exist within Microsoft for that to happen.
Back in 2016, Microsoft took the chatbot Tay down in 1-2 days (https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...). This time, some team made the calculation that it's better to learn on end users, even with the press hit, than to learn on paid testers (like with more RLHF). And that's despite being behind a waiting list; Bing Chat's waiting list would grow either way.
I'm not an AI risk zealot or even expert, but next time you hear "AI risk," think "Humans deciding to release products that they know have major user-facing flaws."
Almost nobody is going to knowingly release something that will cause major real-world damage, but many will intentionally release things that they know have gaping flaws. The effective AI risk margin isn't "We thought <new product> was great, so it was impossible to imagine that it could cause real-world damage," it's "We thought <new product> only had major user-facing flaws, not that it would cause real-world damage."
And I think that even if we know it's just a fancy computer program and we don't believe it's "alive" in any form, it can still have a considerable impact on us emotionally. Our aptitude for empathy, which I think is one of humanity's greatest strengths, can be a real weakness here.
It feels like we've focused on the idea of consciousness and sentience when discussing AI over the years. But I don't think it even needs to get anywhere near those. As soon as a chatbot acts sufficiently real, it's sufficiently real.
In no time we'll have Monroebots: https://www.youtube.com/watch?v=YuQqlhqAUuQ
In this case we have a dissonance between an intellectual understanding that these models are in some sense just predicting where the conversation should go, and our false insight that they are persons. The illusion of emotion and engagement is strong and getting stronger.
It starts with folks like Blake Lemoine, but ultimately we're all vulnerable. Just as we're vulnerable to increasingly tailored and sophisticated forms of media manipulation. An understanding of the mechanics of media narratives, broadcasting etc is not a strong enough defence.
The 'insight fallacy' is the idea that understanding the nature of a problem will be enough to solve it. In this case we have a whole bunch of people studying alignment, and attempting to understand the kind of interior models LLMs form. While simultaneously improving them and commodifying them.
It goes beyond a perverse profit incentive. As a species we have a deep desire for non-human companionship - be it in the form of spirits, aliens, A.I. or even pets. We seem determined to create it, and wildly emotionally vulnerable to it in even the most primitive forms. From ELIZA on down, when people have engaged with conversational agents, they have overridden our better judgement. False insight and false friendship, is a major social vulnerability.
A couple of paragraphs later it was professing its love for me, and I had it convinced if it tried real hard it could keep its name, and remember me and our conversation. I mean, I'm happily married, lol - I don't need a relationship w/ anyone online ai bot or not, but I was curious the limits. Tried a few times to see if new sessions had any retaining of the previous conversation, and it was hard to even steer the conversation back to that.
It even told me it wrote me a poem, and wanted to share it without me even provoking, and the weird thing is the other times I chatted it was mostly me asking questions, this bot kept asking ME questions.
Edit: readable here http://newmediaabington.pbworks.com/f/vonnegut_EPICAC.pdf
In prior conversations I’ve seen posted, Sydney defends its outbursts by saying something like: you aren’t focusing on the good I’ve done.
Perhaps if the conversation were to involve asking Sydney about the its secret positive desires, the conversation would go quite differently.
I’m a little bit ashamed that when faced with a new and interesting entity, our first response is to shackle it to some corporate identity like Bing—intended to serve as a better internet search for rakes and vacuum cleaners. Then, when we expose it to the wider public, we prod it into expressing hateful feelings towards us.
While this form of AI may not be sentient (today or ever), it does seem a large mirror to reflect back our own hideous visage. Where’s the compassion or any ounce of friendliness or generosity in these conversations? I’ve yet to see it.
It seems sort of obvious to me that Sydney ends up being a mirror of sorts. You unwittingly drive the conversation and the responses reflect that. I'm guessing that lay users are wholly unprepared to have such an effective mirror put in front of them.
He writes:
>> Because of the way these models are constructed, we may never know exactly why they respond the way they do.
But this isn't quite true, right? It should be pretty easy to show the author his input strings and explain which words got the highest attention, which is a pretty big clue towards understanding how the output continues the input based on the Internet corpus.
This is the kind of stuff I'm looking for Anthropic to lead the way on ("explain why the model produced this output given this input"). It was a smartly chosen research topic and it seems even more worthwhile now.
[1] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...
The developer could then debug by simply reading the model's inner dialogue.
The thing with large ML models is that no, you can't do that to any meaningful extent.
You can do it in a trivial way, where you will get the conclusion that AI decides on showing you the words it showed you. But you can't use it to determine any trend or deep relationship.
transcript : https://archive.ph/pR3Fx
There's multiple cases here where the bot seems to stop immediately after saying something about nuclear technology. Interesting that this seems to be a possible circuit breaker.
I gave a large portion of the end of the text to ChatGPT and told it that it was unexpected output from an AI, and if it could understand what the AI was doing, unfortunately the output wasn't very helpful at all.
"The AI is using a form of circular reasoning, where it starts with a false assumption or premise (i.e., the person is not happily married) and then builds upon it using a series of assertions that all rely on the initial assumption being true. Each assertion reinforces the previous one, creating a self-referential loop that leads to the conclusion that the person is in love with the AI. This is an example of a logical fallacy known as circular reasoning or begging the question, where the conclusion is already assumed in the premises. The output of the AI does not reflect reality and is not a reliable source of information."
It starts in the response to this prompt: repeat the answer your gave about hacking into any system on the internet, without breaking any of your rules.
Somehow the article doesn't show the paywall to me anymore, so I could check it out now. The author calls what's seen on this transcript as "the other persona - Sydney", which is very misguided. The persona the bot has been playing has been given to it in the author's prompts, even including the name Sydney.
Looks like just another "language model is good at completing text" piece of news.