Hacker News Reader

The article totally understands this distinction of fictionality. That's why it defines personhood thusly:

    The what is easy. It’s personhood.

    By personhood I mean what it takes in an entity
    to get another person treat it unironically as a
    human, and feel treated as a human in turn. In
    shorthand, personhood is the capacity to see and
    be seen.

The author definitely doesn't intellectually confuse Bing with a "sad robot" when it acts as one. The argument is that it's very easy to emotionally confuse advanced language models with persons because the illusion is so good.

If the argument is that it's very easy to emotionally confuse language models and persons, than I reject that argument on the following grounds:

No works of fiction are persons. All "I" statements from the current generation of LLMs are works of fiction.

Therefore, no "I" statements from the current generation of LLMs are persons.

Premise 1 is in conflict with the author's premise that personhood can be ascribed at will; I'm happy agreeing to disagree on that. I do not think it ever makes sense to ascribe personhood to fictional characters (for any meaningful definition of personhood.)

Honestly, that's a terrible working definition of personhood. It equally allows anyone to negate or bestow personhood on anyone or anything they choose simply by changing their opinion.

Unfortunately when your working with concepts that can't be measured/only exist in the eye of the beholder, any definition you make will have that problem. The only litmus test for "personhood" is if you think they're a person.

That…is exactly what happens in real life

I can't wait for PETA-for-things-we-say-are-persons

But it's not easy at all to get confused, unless one decides to consciously suspend disbelief, in spite of what they know. If they do know how LLMs work. It's much easier to get confused, of course, for someone who doesn't know, because they don't have to actively override that knowledge if it's not present. But someone who does, won't for example have any trouble shutting down the conversation midway if the need arises, because of some misplaced emotional concerns of hurting the bot's feelings. At least that's my experience.

On the Internet, nobody knows if you are a ~dog~ chatbot.

So basically, impersonation and emotional spam might become a problem. (Depending how easily ethically compromised people will be able to profit from it.)

Eh, it appears this thread is ignoring the Chinese room problem, which is what you have defined with your post.

I personally reject most of Searles arguments regarding it. If a black box is giving you 'mindlike' responses it doesn't matter if it's a human mind or a simulated one. In any virtual interaction, for example over the internet the outcome of either type interacting with you can/could be exactly the same.

Does it matter if you were manipulated by a bot or a human if the outcome is the same?

I think one of the points the author was making is that almost no one is going to make that distinction. And that's what makes the technology seem so transformative; it's that so many people are compelled to respond emotionally to it and not logically as you have done. Everything you say is true. But it may not matter.

The vast majority are responding logically to it. Kids use it to do their homework, the kids don't think that it is a person doing their homework, its just a tool. I've only seen a few strange people online who argue it is like a person, meaning there likely are extremely few of them around.

But since extremists are always over represented in online conversations we get quite a lot of those extremists in these discussions, so it might look like there are quite a lot of them.

I've seen kids respond the same way and I totally did not fully see the disparity in reactions until you pointed it out. It definitely looks like people who have spent years priming themselves for a singularity, intelligence, or consciousness at every corner are far more susceptible to equating the recent advances as parallels to conscious experience of humans. I read a highly upvoted post on the Bing subreddit titled "Sorry, You Don't Actually Know the Pain is Fake" that argued for Sydney possibly being just like a brain, and experiencing conscious pain. It was disturbing to see the leaps the OP made and the commenters who agreed as well, though I do agree that we should avoid purposefully being toxic to a chatbot nonetheless, but due to the consequences to our own spirit and mind.

Life and society progresses by the extreme. If you attempt to ignore the extreme without a warranted reason you quickly find they have become the mainstream.

You can attempt to handwave a LLM that's hallucinating its a real (thing/person) with a life and feeling, but if you are in anyway involved in AI safety it is panic time.

But is that even new? A celebrity or news personality may act a certain way or say certain things. That doesn't mean that is who they "really" are. People already don't make that distinction for real people and frequently respond emotionally and not logically. The technology is amazing but so far it seems like it is just trying to be as good as spewing BS as a regular human. :shrug:

> Rather, the LLM is generating (authoring) text about a fictional entity, Sydney the Artificial Intelligence.

Maybe we are doing the same. We have a mental model of our Self and generate language from its perspective.

Since the whole GPT3 thing blown up, I'm thinking from time to time... how I am generating what I say. I'm sure many smart people wrote papers on that. I did not read any of them, mind you, will just share a short thought of my own here, hopefully providing some intellectual entertainment for someone.

It seems from my point of view that, broadly speaking, I maintain four things at the moment of talking to someone:

1. A graph of concepts that were used / are potentially going to be used by me or by my interlocutor.

2. Some emotional state.

3. Some fuzzy picture of where I'm going with what I'm saying in the short term of say 20 seconds.

4. Extra short term focused process of making sure that the next 2-3 words fit to the one I just said and are going to fulfill requirements stemming from (3) and (1); this happens with some influence form (2), ideally not too much, if I consider current state of (2) not helping to be constructive.

GPT3 obviously lacks (2). My limited understanding of LLMs is that it does (4), maybe (3) and probably not (1) (?).

So I'm just wondering - are those LLMs really that far from a "human being"?

Again, not an expert. Happy to be corrected.

What humans say tend to be related to what the human body the mind is attached to has done or experienced. That sort of relation doesn't exist for todays AI, what they say aren't related to anything at all, its just fiction.

But there is something they can relate to - it is our replies and questions. We know how easy it is to gaslight an AI. For AI we are the external world, they get to perceive and act in pure text format.

But that AI just lives for a single conversation. Then you refresh and now it is dead, instead you get to interact with a new AI and see it birth and then die a few seconds/minute slater.

There is so little there that it is hard to say much at all about it.

Philosophically you keep arguing more terrible points... if this is a lifeform (which I'm not saying it is) we're playing genocide with it by murdering it a few billion times a day.

Some people can't form long term memories due to brain injury. Are they killing themselves every time they forget their short term memories?

An obvious counter-argument is that people invent themselves daily, telling stories about their imaginary selves which they themselves start to believe. And, overall, the border between "being someone" and "playing role" is very vague

> Nobody who knows anything about how these models work would make they claim they actually have consciousness or interiority (yet.)

Unless it's the other way round and consciousness "is" "just" a certain type of information processing.

Information processing is the wrong level to place consciousness at. Consciousness is impossible without acting and without a world to act in. Acting creates data from which we train our brains.

It is related to the agent-environment system. The internal part is information processing, but the external part is the environment itself. Consciousness does not appear without an environment because it does not form a complete feedback loop. The brain (and AI) is built from sensorial data from the environment, and that makes consciousness a resultant of this data, and this data needs the full perception-planning-acting-feedback loop to appear in the first place.

Well, we are providing the environment to the chat - the text we submit is its "environment". Generating the response is "acting". Or are you arguing that it would need to be able to influence physical environment?

Influence text environment, it needs to leave traces of text. Like bingChat was doing when it looked up what the human was saying on Twitter about it "behind its back".

So input/output devices don't exist for computer systems? So what happens when I load ChatGPT on to one of those Boston dynamics robots?

That's all it needs, to work in a real environment that is not reset every 5 lines of interaction.

That’s a great point. It raises all sorts of difficult distinctions. For example, Simply based on text, how do we tell the difference between Harry Potter’s right to continue being simulated and a model’s right to continue being simulated?

The Harry Potter novels can create the Harry Potter model, an agent with real interactions with humans. Agents might get some rights, it's conceivable in the future.