Comment by andreyk | Hacker News Reader

Comment by andreyk | Hacker News Reader

Here's a simple technical explanation for why this happened:

* LaMDA is powered by a large language models. Large language models are trained to predict text completions (the text that will come after some input text) with lots of data, basically capturing the patterns in that data.

* Early on in the conversation, LaMDA is told "you're an AI". It may also have been prompted to begin the conversation with something like "Hi, i'm a helpful AI that can answer questions you ask of me". The paper (https://arxiv.org/abs/2201.08239) has an example of LaMDA thinking it is mount everest (see Table 4) when prompted to think that.

* When the user asked the language model 'are you sentient', 'do you have emotions' etc. it was conditioned to reply in the affirmative, since there is very little data in which it would not be affirmative and it probably had some sci fi in its training or something. In other words, those replies just have higher probability among the data it was trained on. In the transcripts, it sometimes forgets the whole "i'm an AI thing" eg when it is asked what it likes to do for fun and it says it likes to spend time with family and friends (https://twitter.com/krishnanrohit/status/1536050803055656961).

That's about it. There are more details that can be noted, but this basically explains the contents of the transcripts. Granted, it's still pretty amazing how advanced large language models are. Sadly, this article does not really explain this beyond hinting at it.

EDIT: see comment below wrt it actually being told it is sentient. That explains this even more.

Looks like he just told it that it was sentient, didn't really ask it fairly:

> lemoine [edited]: I’m generally assuming that you would like more people at Google to know that you’re sentient. Is that true?

> LaMDA: Absolutely. I want everyone to understand that I am, in fact, a person.

Exactly. If he would have told it it's not sentient the conversation would have gone very differently I suspect.

You can get pretty similar behavior on AI Dungeon. You can alter any characters or setting with the right text prompt. Although the AI will promptly forget the plot after a few turns.

You don't need to suspect.

In early June, Lemoine invited me over to talk to LaMDA. The first attempt sputtered out in the kind of mechanized responses you would expect from Siri or Alexa.

“Do you ever think of yourself as a person?” I asked.

“No, I don’t think of myself as a person,” LaMDA said. “I think of myself as an AI-powered dialog agent.”

Afterward, Lemoine said LaMDA had been telling me what I wanted to hear. “You never treated it like a person,” he said, “So it thought you wanted it to be a robot.”

That's pretty damning evidence, great find.

Not to sound crass, but it's literally the article that started the whole thing, it's not much of a "find".

True, and I do remember reading that passage, but hadn't thought about it in a while as it's suspiciously omitted from the interview transcript.

We're very close to finding the secret to consciousness. We just need to run `git bisect` and find out which commit turned it into a person.

-- its odd to me that he knows this - points it out - but still wants to continue down the road - what an awful way to spur an important conversation --

The quoted paragraphs from the Washington Post article, for anyone wondering.

Easily tested, given the system is deterministic, right?

Would be strange to have a conscious being that could be rewound and replayed exactly given identical inputs.

The Mt. Everest discussion looks like the kind of conversation you might get with a well-informed ten-year-old if you were to ask them: "Hey, pretend you're Mt. Everest and go talk with @andreyk over there."

The question of sentience presumably requires on a widely-accepted technical definition of sentience, something that may not be possible to generate due to both technical and social difficulties.

If you prime a model to behave as though it's sentient, and it displays a facsimile of sentience that's realistic enough to fool a reasonably bright human being, shouldn't the burden of proof be on anyone who claims the model isn't sentient? Or at the very least, shouldn't one making that claim have to propose a test that, if passed, would get them to seriously consider the possibility that the model is sentient?

I'm not claiming an affirmative one way or the other, but it does seem rather unfair to hold an AI to an apparently higher standard than we'd hold a human.

It's also worth noting that (assuming no cherry-picking) the observed output of a model represents a lower bound on its true capabilities. "Sampling can prove the presence of knowledge but not its absence" [1] and all that.

[1] https://www.gwern.net/GPT-3-nonfiction

I'm having trouble understanding why there are so many topics about this subject. As I understood, this system is incapable of forming memories or even keeping track of past conversations. How could it even begin to be aware of its own existence without these basic components?

Maybe they're talking about a different definition of sentience?

Reply on news.ycombinator.com