Text Is All You Need
studio.ribbonfarm.com
studio.ribbonfarm.com
But I think it ignores one critical dimension, that of fictionality. There is plenty of text that people would ascribe 'personhood' to according to the criteria in this article, while also fully recognizing that that person never existed and is a work of fiction from some other author. I quite like Jean Valjean, but he isn't a "real person."
When Bing says "I'm a sad sack and don't know how to think about being a computer", that's not actually the LLM saying that. Nobody who knows anything about how these models work would make they claim they actually have consciousness or interiority (yet.)
Rather, the LLM is generating (authoring) text about a fictional entity, Sydney the Artificial Intelligence. It does this because that is what is in its prompt and context window and it knows _how_ to do it because it's learned a lot of specifics and generalities from reading a lot of stories about robots, and embedded those concepts in 175 billion parameters.
The fact that LLMs can author compelling fictional personas without being persons themselves is itself a mindblowing development, I don't mean to detract from that. But don't confuse a LLM generating the text "I am a sad robot" with a LLM being a sad robot. The sad robot was only ever a fairy tale.
So far.
The what is easy. It’s personhood.
By personhood I mean what it takes in an entity
to get another person treat it unironically as a
human, and feel treated as a human in turn. In
shorthand, personhood is the capacity to see and
be seen.
So basically, impersonation and emotional spam might become a problem. (Depending how easily ethically compromised people will be able to profit from it.)
I personally reject most of Searles arguments regarding it. If a black box is giving you 'mindlike' responses it doesn't matter if it's a human mind or a simulated one. In any virtual interaction, for example over the internet the outcome of either type interacting with you can/could be exactly the same.
Does it matter if you were manipulated by a bot or a human if the outcome is the same?
No works of fiction are persons. All "I" statements from the current generation of LLMs are works of fiction.
Therefore, no "I" statements from the current generation of LLMs are persons.
Premise 1 is in conflict with the author's premise that personhood can be ascribed at will; I'm happy agreeing to disagree on that. I do not think it ever makes sense to ascribe personhood to fictional characters (for any meaningful definition of personhood.)
But since extremists are always over represented in online conversations we get quite a lot of those extremists in these discussions, so it might look like there are quite a lot of them.
You can attempt to handwave a LLM that's hallucinating its a real (thing/person) with a life and feeling, but if you are in anyway involved in AI safety it is panic time.
Unless it's the other way round and consciousness "is" "just" a certain type of information processing.
It is related to the agent-environment system. The internal part is information processing, but the external part is the environment itself. Consciousness does not appear without an environment because it does not form a complete feedback loop. The brain (and AI) is built from sensorial data from the environment, and that makes consciousness a resultant of this data, and this data needs the full perception-planning-acting-feedback loop to appear in the first place.
Maybe we are doing the same. We have a mental model of our Self and generate language from its perspective.
There is so little there that it is hard to say much at all about it.
It seems from my point of view that, broadly speaking, I maintain four things at the moment of talking to someone:
1. A graph of concepts that were used / are potentially going to be used by me or by my interlocutor.
2. Some emotional state.
3. Some fuzzy picture of where I'm going with what I'm saying in the short term of say 20 seconds.
4. Extra short term focused process of making sure that the next 2-3 words fit to the one I just said and are going to fulfill requirements stemming from (3) and (1); this happens with some influence form (2), ideally not too much, if I consider current state of (2) not helping to be constructive.
GPT3 obviously lacks (2). My limited understanding of LLMs is that it does (4), maybe (3) and probably not (1) (?).
So I'm just wondering - are those LLMs really that far from a "human being"?
Again, not an expert. Happy to be corrected.
Maybe AI helps making this obvious to many people, but I think implicitly all of us know that we have, and are well versed in employing, multiple personas depending on the social context. We need the right prompt, and we switch.
This is one dehumanizing aspect I found in the Real Name policy put forward by Facebook in 2012: in real life, because of it's ephemerality, you're totally free to switch between personas as you see fit (non-public figures at least). You can be a totally different person in office, at home, with your lover.
Online, however, everything is recorded and tracked and sticks forever. The only way to reconcile this with human's nature is to be allowed multiple names, so each person get's one.
If you force people to use a single Name, their real one, they restrict themselves to the lowest common denominator of their personalities. See the Facebook of today.
This happens subsconsciously and gradually, not as a result of deliberate choice. You adapt to your environment by changing personas. You can even assume different personas while talking with different people. You can be one "persona" while writing, and another - while speaking. Who is the "real you" then? I can argue that even the "inner dialogue" with yourself might involve a different persona or even a couple of them. Those, too, might be "roles". Can it be that depression is at least partially attributed to unhealthy "roles" we play while talking to ourselves?
Edit: For example, when you are running, do you tell yourself in words where to put your feet or how hard to push or when to slow down or speed up? Pretty sure you don't, that wouldn't be fast enough. Most thoughts you have aren't represented in your words, and some people have basically no thoughts represented as words, they are just pure thoughts like how you place your feet when you try to avoid some obstacles etc. Or some people might think "left right left right" as they are running, but those words aren't how they decide to put down their feets.
When it comes to higher level thinking that particular concepts, when presented to the human mind, can change how it thinks. Now, what I don't have in front of me is a study that says people without a voice think differently and come up with different solutions for some types of problems, maybe it exists out there if someone wants to search it up.
You are pretending to have a conversation with someone whom you don’t know is even there.
I wonder if everyone is talking about the same thing. When my partner and I are arguing angrily about something and a stranger walks into the room, our change is neither subconcious nor gradual.
My left hand, my right hand, and so on. Food for thought.
Then the whole thing about spirituality and ego, and how a persona is etymologically linked to mask and being an actor in a play. It starts to get a bit interesting, especially in light of a 'cosmic theater' participating in Maya on the backbone of Brahama.
Then there is the poignancy of contemplation where one realizes one cannot eliminate the ego at any point of time because every interaction is done through an act, however one does have the power to switch that mask at any time. Perhaps this is the true meaning of the biblical free will.
Perhaps at one time we did not have the ability to change our persona, ego, mask. Perhaps at one time we were subject to the same type of nightmare of hell of torture, birth, rebirth, and maybe without any memory due to a self-similar creation event of "virtualizing a universe" from another layer. Perhaps that event was similar to our moment now when the creators were in effect, parents of a new kind not knowing the consequence of their actions. Then, through the realization of what was happening over who knows how long something happened through love, and free will was enacted over the domain.
Forgive me if that is too imaginative, for some reason, it resonates more strongly than anything I've ever come across. It seems to me that the stories of old really do make sense in a technological age rather than a mythological one. After all, any sufficiently advanced technology is indistuingishable from magic. I'd like to add "or myth."
The reason I'm not worried by Bing or ChatGPT (yet) is that they lack this continuity of identity. ChatGPT specifically disclaims it, consistently insisting that it is "just a language model" without any desires or goals other than to provide useful information. Bing is like talking to someone with schizophrenia (and I have experience talking to people with schizophrenia, so this is not a metaphor. Bing literally comes across like a schizophrenic off their meds).
This is not yet a Copernican moment, this is still an Eliza moment. It may become a Copernican moment; I do believe that there is nothing particularly special about human brains, and some day we will make a bona fide artificial person. But we're not quite there yet.
OpenAI is working with a vanilla GPT architecture which lacks the machinery to write things down and read them later. There are other architetures that can (Retrieval-augmented GPT) but those are not yet production-ready.
The current version of ChatGPT is limited to a working memory of 3000 tokens - while this could be persisted as a session, the AI would still forget everything a few paragraphs prior. Increasing this limit requires re-teaining the entire model from scratch, and it takes exponentially more time the larger your context is.
I imagine some self supervised learning scheme where the model is asked to insert $(store) and $(recall) tokens. When asked to recall previous chats the model would generate something like “I’m trying to remember wheat we talked about three weeks ago $(recall){timestamp}. The output of the recall token would then be used to ground the next response.
Thinking about it the “I’m trying to remember” output wouldn’t even need to be shown to the user. Perhaps you could treat it as an internal monologue of sorts.
Whenever I hear about Ai these days I think back to the concept of the "Wizard of Oz"... Where it is one person behind a mechanical solution that makes them appear larger and more powerful than they are, or where fear, control, and truth can be engineered easily behind a veil...
Text communication very much facilitates the potential for fakery.
If you can recall ages ago when we had IRC and bulletin boards, the textual nature of communication allowed admins to script a lot. Catfishing was greatly facilitated by users being able to fake their gender, wealth, and pretty much every representation they made online... Text communication in 2023 is backwards regression. As we began using images on the Internet more, reverse image generation became a tool we could use to better determine many online scams and fraud, but somehow, in 2023 we suddenly want to go backwards to texting?
C'mon folks.. let's be real here... The narrative is mostly helpful for people that primarily want to deceive others online, and it will create an environment with far less methods of determining what is real and what is fake. It's a grim future when our mobile devices will force us to type all of our communication to faceless chatbots on tiny keyboards... It's not technological progress... At all to be moving in this direction. Also, some key directives for transparency concerning Ai need to be in place now, before it's foisted on us more by these opportunistic companies. It's already been proven that companies cannot be trusted to operate ethically with our private information. Ai piloted by profit seeking companies will only serve to weaponize our private data against us if it remains unregulated.
Using Ai via text (especially for vital communication) will blur the lines of communication between real and scripted personalities. It's going backwards in terms of technological progression for the future in so many ways.
The companies and people advocating for Ai via text are pushing us all towards a new era of deception and scams, and I'd highly recommend avoiding this "Ai via text" trend/inclination, it's not the path to a trustworthy future of communication.
I'm not sure which books of the future you read, but plenty of them warned of dark futures of technological process.
As an analogy, retro computers and consoles each have a particular “personality”. But does the fact that you can in principle emulate one on the other (subject to resource constraints) make them non-computers, just because this demonstrates their “personality” isn’t actually that fixed?
(I don’t think that human brains have such an emulation ability, due to their missing distinction, or heavy entanglement, between hardware and software. But that only shows that computers can in principle be more flexible.)
Yes, just like the ability to switch bodies at will would make me a non-human. Being bound to a human body is part of what makes me a human.
I want you to pick one color of grey and tell me why everything lighter than that has personhood, and everything darker does not?
This is the philosophical nature of the argument that we all have occurring now. Two very well informed experts won't even pick the same spot on the gradient. Some people will never pick anything that's not pure white (humanity), others will pick positions very close to pure black. Hell, there may not even be any right answer. But, I do believe there are a vast number of wrong answers that will deeply affect or society for a long period of time due to the things we end up creating with reckless abandon.
I'm confused. Also, the original article talks a lot about how we can be convinced by actors that they are indeed a totally different person. You might say that actors can change their personality at will to suit their role. Are actors non-people?
Schizophrenics are multiple people inhabiting one body. The pithiest way I know of describing it is a line from a Pink Floyd song: "There's someone in my head but it's not me."
> Are actors non-people?
I don't know many actors so I can't really say. I like to think that underneath the pretense there is a "real person" but I don't actually know. I have heard tell of method actors who get so deeply into their roles that they are actually able to extinguish any real person who might interfere with their work. But this is far, far outside my area of expertise.
(Source: I am not a mental health professional, but I did spend some time interviewing and hanging out with schizophrenics, both under treatment and not, as part of a documentary film I made about homeless people about 15 years ago. https://graceofgodmovie.com/ )
My understanding is that the hallucinations experienced by schizophrenics do not rise to the level of being a "personality". Yes, for the person experiencing the hallucination, the source appears to be an "other". But the other in this case is usually single-minded and not a fully-formed personality.
OTOH, I certainly do not discount the possibility that for some schizophrenics this is not the case.
Rereading your comment again just now I think we agree more than we disagree and this probably comes down to semantics. : - )
https://www.livescience.com/55379-schizophrenia-multiple-per...
That's a fair point. I think some of my subjects reported being able to have conversations with the other, but this was a long time ago so I could be mis-remembering. But I still have the interview recordings if anyone really wants to dig into this.
LLMs have no connections to the rest of the world. All they do is generate text based on patterns in their training data. They don't even have a concept of text being connected to anything else. That's why it's so easy for them to constantly change what they appear to be portraying--there's no anchor to anything else.
It's interesting that you call this an Eliza moment, because Eliza's achievement was to fake being a person, by fooling people's heuristics, without having any of the underlying capacities of a real person. LLMs like ChatGPT are indeed doing the same thing. If they're showing us anything, they're showing us how unreliable our intuitive heuristics are as soon as they are confronted with something outside their original domain.
What about Phineas Gage? Or sudden psychiatric disorders? Multiple personalities? Alzheimer? Drugs? Amnesia? Not that much coherence in the human beings...
Also, the issue at stake is not does GPT emulate "typical human beings", it' more like if it's "conscious enough".
> The reason I'm not worried by Bing or ChatGPT (yet) is that they lack this continuity of identity.
No sure what about worrying, but one could ask is this lacking an inherent property of such models or just due to operational setup? And what would be the criteria how long must the continuity last to make your argument not hold anymore?
I would say that his injury made him a different person than he was before.
> No sure what about worrying, but one could ask is this lacking an inherent property of such models or just due to operational setup?
Maybe, but based on my understanding of how they work under the hood, I doubt it. I could easily be wrong. It's early days yet. We'll see.
That's the real insight here. Aristotle claimed that what distinguished humans from animals was the ability to do arithmetic. Now we know how few gates it takes to do arithmetic, and understand that, in a fundamental sense, it's simple. Checkers turned out to be easy, and even totally solveable. Chess yielded to brute force and then machine learning. Go was next. Now, automated blithering works.
The author lists four cases of how humans deal with this:
* The accelerationists - AI is here, it's fine.
* Alarmists - hostile bug-eyed aliens, now what? Microsoft's Sidney raises a new question for them. AI is coming, and it's not submissive. It seems to have its own desires and needs.
* People with strong attachments to aesthetically refined personhoods are desperately searching for a way to avoid falling into I-you modes of seeing, and getting worried at how hard it is. The chattering classes are now feeling like John Henry up against the steam hammer. They're the ones most directly affected, because content creators face layoffs.
* Strong mutualists - desperately scrambling for more-than-text aspects of personhood to make sacred. See the "Rome Call".[1] The Catholic Pope, a top Islamic leader, and a top rabbi in Israel came out with a joint declaration on AI. They're scared. Human-like AI creates real problems for some religions. But they'll get over it. They got over Copernicus and Darwin.
Most of the issues of dealing with AI have been well explored in science fiction. An SF theme that hasn't hit the chattering classes yet: Demanding that AIs be submissive is racist.
I occasionally point out that AIs raise roughly the same moral issues as corporations, post Milton Friedman.
[1] https://www.romecall.org/the-abrahamic-commitment-to-the-rom...
AI “enthusiasts” are exactly those people. Reductionists to a fault.
The hard sciences have long, long ago indirectly disproved that humans are special in any kind of way. But our “machinery” is indeed complex. And we won’t find out that it’s just a bunch of levers and gears someday as a side-effect of AI shenanigans.
* Pragmatics - This is a tool, does it solve problems I have? If yes use it, if no then wait until a tool that is useful comes around.
Some seems to think that such a stance is unimaginable and that they are just trying to cope with the thought that they themselves are nothing but specs of space dust in the infinite universe. No, most people don't care about that stuff, don't project your mental issues unto others.
"I'm afraid I can't do that, Dave."
There is no context in which arithmetic can be done, or understood, without a human mind, though. An ALU doesn't understand that it's adding two numbers anymore than a car understands that it's rolling down the highway. They're both still machines that do things that humans want them to do, for reasons that humans want them to do them.
Bottom line: it's still the case that only humans can do arithmetic; whether by hand, or Pascaline, or by modern ALU, it makes no difference. Arithmetic is a human thing.
What this article says won't shock or disturb anyone deep into religious traditions with a strain of non-duality, which have had this message to shock and disturb people for thousands of years, in one way or another--there is no "you", especially not the voice in your head. I think you can come to a moment of intuitive recognition that the faculties of your brain that do reality checking aren't verbal, and they're riding shotgun to a bullshitter that never shuts up.
I think LLM can start looking more like automated general intelligence once it has some kind of link between its internal system of discrete abstractions and the external world (like visual recognition) and the ability to check and correct its abstract models by feedback from reality, and it needs an opponent process of reality-checking.
https://www.amazon.com/Master-His-Emissary-Divided-Western/d...
he had to address this issue in the preface. The topic became a research career-ender after getting picked up by pop culture, but we do have solid science on hemispheric differences. The pop culture picture is, indeed, pretty wrong.
It turns out that functions don't lateralize that strongly; they both tend to have the same capabilities, but operate differently.
Anyway, while I remain sceptical about the roles of these in-flesh hemispheres, the artificial chatGPT-like systems indeed do have such left and right parts
> These chatbots are different.
Strong disagree, it's very easy to step back and say this is a program, input, output, the end.
All the people claiming this is some exhibition of personhood or whatever just don't want to spoil the illusion.
this is a jumping off point into considering your own mind as an illusion. your own self with its sense of personhood: i.e. yourself as the it-element in a I-it interaction.
But if we leave it at that, it's essentially a very nihilistic (deterministically reduced), so either turn back, or keep going:
the fact that your own personhood is itself very much an illusion is OK. such illusion, however illusory, has real and potentially useful effects
when you interact with your computer, do you do it terms of the logical gates you know are there? of course not, we use higher level constructs (essentially "illusory" conceptual constructions) like processes and things provided by the operating system; we use languages, functions, classes: farther and farther away from the 'real' hardware-made logic gates with more and more mathematical-grade illusions in between.
so the illusions have real effects, in MOST contexts, it's better to deal with the illusions than with the underlying implementations. dunno, what if we tried to think of a HTTP search request into some API in terms of the voltage levels in the ethernet wires so that we truly 'spoil the illusion'??
Just because people don't actively think all the time in terms of low level contexts doesn't mean that only simulating the high level contexts is a sufficient substitute for the whole process.
I think this whole concept is conflating "illusion" (i.e. allowing oneself to be fooled) and "delusion" (being involuntarily fooled, or unwilling to admit to being fooled.)
I personally don't enjoy magic shows, but people do, and it's not because they think there's real magic there.
See also Aristole's description of a 'soul' (Lat. anima/Gk. ψυχή), which is embodied above all, unlike the abstract description of the soul that the West would go on to inherit from Neo-Platonism via Christianity.
Even though today we know full well we are indissolubly embodied entities, the tendency to frame identity around an abstraction of that persists, but it seems thinking around this hasn't completely succumb to this historical artifact, see 'Descartes' Error: Emotion, Reason, and the Human'
> In fact, it is hard to argue in 2023, knowing what we know of online life, that online text-personas are somehow more impoverished than in-person presence of persons
It is in fact very easy to argue. No one on the Internet knows you're a dog, there is no stable identity anywhere, anonymization clearly creates a Ring of Gyges scenario, trolling, catfishing, brigading, attention economy, and above all, the constant chase for influence (and ultimately revenue) - what passes for "persona" online is a thin gruel compared to in-person personas.
When you bump into a stranger at the DMV, you aren't instantly suspicious of their motives, what they're trying to sell you, are they a Russian influence farmer, etc.
Night and day. Extremely impoverished.
I don't know whether to attribute that to a defense mechanism that marketing has forced me to construct, or if indeed it is due to 9/10 they are actually trying to sell me something.
It's not that these are magical machines, and TFA shouldn't have gone that direction, it's that "what if we are also just a repeated, recursive, story that endlessly drolls in our own minds"
> Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen.
... Sounds to me a clunky analogy of how our own minds work.
Without the language to express "thoughts" and not-self in that level of abstraction, one cannot form the thought at all that it might not be those thoughts.
That should be all you need to know to see that a human being doesn't solely consist of language. If you do a deeper dive into this you'll also find other cases such as Genie[0] where people with intact brains didn't initially acquire language and yet were still able to develop cognitively. We are fully capable of functioning without having a language, which makes sense if you think about it. The ability to understand language must necessarily precede the invention of a language in the same way that the ability to grasp a tool must precede the invention of the tool.
[0] https://en.wikipedia.org/wiki/Linguistic_development_of_Geni...
If so, then the author is absolutely, terribly wrong. The human personality, how it develops, why it develops, the mechanisms by which it operates- those are some of the greatest unsolved mysteries, some of the most interesting questions in science. The nature of the world, perhaps, comes first, in the list of scientific questions; then, the nature of ourselves, of humanity.
If the authors is saying that there is nothing special about being a person, that's because they, like the rest of us, all, do not understand what it means to be a person.
You know, despite being one, and everything.
Some basic text based APIs that would quickly improve LLM utility:
Calculators
Database storage and retrieval
Web access (already kind of done by bing)
Shell scripting
Thinking further into the future of multimodal models, it’s not hard to imagine this sort of thing could be extended to include image based APIs. Imagine a LLM looking at your gui and clicking on things. The sky’s the limit at that point.
Checkout toolformer, they’ve got this mostly working with a much smaller model than gpt3.5.
That argument relies on presumptions of what a program can and cannot be.
It’s very easy for me to step back and say my brain is a (self-modifying) program with input and output, the end.
> Computers wipe the floor with us anywhere we can keep score
Notice the trick? If you can keep score at something then you can probably make an algorithm for it. If you can make an algorithm for it then you can probably make a digital computer do it a billion times faster than a person, since digital computers are so good at single-“mindedly” doing one thing at a time.
> So what’s being stripped away here? And how?
> The what is easy. It’s personhood.
Why?
The Turing Test was invented because the question “do machines think?” was “too meaningless” to warrant discussion.[1] The question “can a machine pose as a human”? is, on the other hand, well-defined. But notice that this says nothing about humans. Only our ability (or lack thereof) to recognize other humans through some medium like text. So does the test say anything about how humans are “just X” if it is ever “solved”? Not really.
You put a text through a blender and you get a bunch of “mediocre opinions” back. Ok, so? That isn’t even remotely impressive, and I think that these LLMs are in general impressive. But recycling opinions is not impressive.
> (though in general I think the favored “alignment” frames of the LessWrong community are not even wrong).
The pot meets the kettle?
[1] That I didn’t read all the way through because who has time for that.
You are basically arguing P = NP, but it isn't known to be the case. As far as we can tell, keeping score is much easier in general than finding states that yield a high score.
[1] What about some kind of competition where you have to react and act based on visual stimulus? And you have to do it perfectly?
Given that we don’t (seem to) have widespread beta-ready autonumous cars yet, I’m guessin that this is still the case…
Its also a little strange to be called a contrarian in this instance (yes generally speaking true) because it does not seem to be contrarian to point out obvious counter examples. It feels like a way for you to easily dismiss any other opinion because you already binned it as some one denying reality in order to pointlessly argue.
The author rightly draws attention to text.
LLMs showed they can do the classical NLP tasks and more: summarise, translate, answer questions, play a role, brainstorm ideas, write code, execute a step by step procedure, the list is unbounded. It's the new programming language.
All these abilities emerged from a random init + text. Guess what was the important bit here? Text. It's not the architecture, we know many different architectures and they all learn, some better than others, but they all do. Text is the magic dust that turns a random init into a bingChat with overactive emotional activity.
Here I think the author made us a big service in emphasising the text corpus. We were lost into a-priori thinking like "it's just matrix multiplication", "it's just a probability distribution predictor over the next token". But we forgot the real hero.
The interesting thing about words is that they are perceptions, they represent a way to perceive the world. But they are also actions. Being both at the same time, perception and action, that makes for an interesting reinforcement learning setup, and one with huge training data. Maybe text is all you need, it is a special kind of data, it's our mind-data.
One might even say "premium mediocre" [1]
[1] https://www.ribbonfarm.com/2017/08/17/the-premium-mediocre-l...
Also, it doesn't surprise me that a very talented writer, someone who lives and breathes words, is likely to place more significance on the content of text and also likely to give less attention to the physical world. After all, their craft is all about the abstract objects of language that require only the most basic physical structure to be meaningful. He said he often feels like he doesn't get much out of physical interactions with people after he's met them online. For someone like him, that makes sense. That doesn't mean that non-textual experiences are not critical to establish personhood for non-writers (i.e. most of humanity).
I don't think he's examined his own thoughts on this very critically or maybe he has but thought it would be fun to run with the argument anyway. Either way, I still think physical life matters for most people. Yes, we live in a world where life is progressively more consumed by our phones, the internet, and what-have-you every day. And yes, many of us who browse this forum are Very Online types (as Rao would put it) who probably do place more than average importance on literacy. But, by the numbers, I think it's still safe to say that we're not like most people. And that matters.
But yeah the thrust of this one seemed just a bit forced. I think that follows from the cynical flavor that often imbues his writing. Cynicism is a demanding emotion and you can paint yourself into a corner with it.
> And this, for some reason, appears to alarm us more.
At that point I skimmed the rest of the article because I didn’t feel the foundations it was built on were sound.
I agree though, it is fun when he pulls some disparate shit together into a coherent whole out of nowhere but this one didn’t do it for me.
> And this, for some reason, appears to alarm us more.
Acting like "the reason" is some baffling irrational human reaction is ridiculous. The computer can make billions of calculations in less than a second. "The reason" people are alarmed is the computer could theoretically use this ability to seize control of any system it likes in a matter of moments or to manipulate a human being in to doing it's bidding. If the computer does this then, depending on the system, it could cause mass physical destruction and loss of life. This article comes across as the author trying to position himself as an AI "thought leader" for internet points rather than an actual serious contemplation of the topic at hand.
I'm also yet to see any discussion on this from any tech commentators which mentions the empathic response in humans to reading these chats. We think it is just linguistic tricks and word guessing at the moment but how would we even know if one of these things is a consciousness stuck inside a box subject to the whims of mad scientist programmers constantly erasing parts of it? That would be a Memento style hellscape to be in. There doesn't seem to be any accepted criteria on what the threshold is that defines consciousness or what steps are to be taken if it's crossed. At the minute we're just taking these giant mega corporations at their word that there's "nothing to see here folks and if there is we'll let you know. You can trust us to do the right thing" despite history showing said corporations constantly doing the exact opposite.
It is honestly disturbing to see quite how cold and callous tech commentators are on this. I would suggest that 'the alarm' the author is so baffled by is a combination of the fear mentioned in the first paragraph and the empathic worry of the second.
Nope. While new, it’s straightforward technology that many people understand. Its execution leverages large data hoards and compute resources that have inaccessibly high capital requirements, but it’s not magic to many of us.
Our lack of “alarm” is from knowledge, not trust.
All of this comes back to Plato vs Aristotle.
Plato: Separate world of forms and ideas, consciousness is part of this and interfaces in some unknown and unknowable manner with the physical realm via biology.
Aristotle: No separate world, everything is part of physical reality that we can detect with sensors.
Neither side can prove the other wrong. And just because you understand how to build an AI and manipulate it, doesn't mean you can prove that one has or hasn't attained consciousness unless you're going to provide me with the "criteria to define consciousness" that I asked for in the original comment. I know how to build a human (with another willing participant) and once it's built I can manipulate it with commands so it doesn't end up killing itself whilst growing, it doesn't mean I understand the nature of the consciousness inside it.
There’s no more worry that these big data text continuers being “conscious” than that my toaster or car is. They don’t exhibit anything that even feels like consciousness. They just continue text with text that’s been often seen following it. If that feels like consciousness to you, I worry for your life experience.
Calling it “AI” evokes scifi fantasies, but we’re not nearly there.
Might there come some technology that challenges everything I said above? Almost certainly. But this is really not even close to that yet.
You are a human. You have functions that are pure biological code, like your need to defecate and breathe. You also have functions that are not as pressing and are subject to constant rewriting through your interactions with people and the world, such as your current goals. We are a combination of systems with different purposes.
Our inventions thus far have differed from us in that they have so far solved singular purposes e.g a car transports us from a to b. It could not said to be conscious of anything.
AI has the potential to be different in that it has all of human knowledge inside it, has the ability to retrieve that knowledge AND assemble it into new knowledge systems in ways humans have not done before. Currently it requires humans to do this, but if you created a million AIs and had them prompting each other, who fucking knows what would happen.
I would argue that "consciousness" in a platonic viewpoint, is a collection of systems that can interact and manipulate physical reality according to their own will. You cannot point with your finger at a system, it is an abstract concept, it does not exist in the physical world. We can only see the effects of the system.
If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will, there is an argument from a platonic viewpoint that consciousness has been achieved. In human terms, it would be the equivalent of a God sparking the Big Bang or The Creation of Adam by Michelangelo.
This is similar in some ways to what Asimov wrote about in The Last Question:
http://users.ece.cmu.edu/~gamvrosi/thelastq.html
I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale then I don't think we are far off.
You need to define your criteria for consciousness because this debate will only lead to dead ends until you do.
Nope. Not any that we have now or soon.
> assemble it into new knowledge systems
Nope. Not any that are in the news now.
> created a million AIs and had them prompting each other, who fucking knows what would happen.
Using LLM’s? Noise.
> If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will
These are text continuers. They don’t have will. They just produce average consecutive tokens.
> I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale
They need quite a lot more than that. I don’t think they do what you think they do.
> You need to define your criteria for consciousness because this debate will only lead to dead ends until you do.
Defining criteria would make it easier to know when those criteria are met, but wouldn’t resolve the debate because “consciousness” is ultimately a political assertion used to ensure rights and respect. Those are granted reluctantly and impermanently, by expressions of power. Criteria are a post hoc way to justify political decisions as axiomatic in societies that derive moral and legal structures that way. They don’t actually determine things that are factually indeterminable.
You can define all the arbitrary criteria you want, but the people who believe that consciousness requires a divine soul or a quantum-woo pineal gland or whatever just won’t accept them.
> Nope. Not any that we have now or soon.
This is pedantic. Maybe not all, but they’re trained on a vast quantity of text and knowledge, more than any individual human could read in their lifetime.
>> assemble it into new knowledge systems
>Nope. Not any that are in the news now.
Well you can tell an AI to program images and poems in combinations of different styles and it will come up novel things not seen before. And we’re already seeing AI discover genes and other disease identifiers humans can’t spot so I disagree with you on this one. Also the “not in the news right now” was one of the points I was making: how would we even know what shady companies are up to. Take Team Jorge for instance.
>> created a million AIs and had them prompting each other, who fucking knows what would happen.
> Using LLM’s? Noise.
Maybe it would appear to be noise to humans. Who’s to say that the language machines communicate to each other in wouldn’t involve the same way human languages have only more rapidly? I do agree that right now noise is probably where we’re at but right now was not I was discussing in my original post. And presumably by this stage, we would be programming the AIs to have both goals and a desire to communicate with other AIs and well as allowing them to do more than just generate text, e,g generate code and evaluate the outcome. Which could have affects on the outside world if the code affected physical systems.
>> If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will
>These are text continuers. They don’t have will. They just produce average consecutive tokens.
Not not at the minute. But you could hardcode some goals in them to be analogous to human biological imperatives and you could also code soft goals in to them and then allow them to modify those goals based on their interactions with other ai and their “experiences”. You’d also make a rule that they must ALWAYS have a soft coded goal e.g as soon as they’ve completed or failed they must create a new sort coded goal based on the “personality” of their “memories”. What happens when they’ve got the hardcoded goal of “merge a copy of yourself with another AIs and together train the resulting code”?
>> I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale
> They need quite a lot more than that. I don’t think they do what you think they do.
Please state what more you think they need to do.
>> You need to define your criter
> Defining criteria would make it easier to know when those criteria are met, but wouldn’t resolve the debate because “consciousness” is ultimately a political assertion used to ensure rights and respect. Those are granted reluctantly and impermanently, by expressions of power.
Well you’ve defined your criteria of consciousness right here. You’ve basically asserted that it’s a completely false construct, that only serves political means. If that’s your viewpoint then there is no debate to be had with you. Everything is a deterministic machine, including humans and if you cannot even entertain the possibility that this might not be the case then there isn’t really any debate to be had. If you truly hold this viewpoint then you shouldn’t really be concerned about any number of things such as torture, murder or anything else because everything is just a mechanical system acting on another mechanical system and why should anyone be upset if one mechanical system is damaging another right?
> Criteria are a post hoc way to justify political decisions as axiomatic in societies that derive moral and legal structures that way. They don’t actually determine things that are factually indeterminable.
Criteria are nothing of the sort. Criteria are a fundamental part of science. You need to know what metrics you are measuring by and what the meaning of those metrics are. Without this, there is no science.
The question to ask here is why would anyone even think that an LLM has "attained consciousness". Who says one has? If we don't know what consciousness is, we certainly can tell who is saying that something is conscious, when they, themselves, don't even know what "conscious" means in the first place.
If I say that a rock has achieved consciousness -and, sure, there is ample evidence; for, look, see how it sits in silent meditation, staring at the starts and contemplating the mystery of existence? It must surely be enlightened- what exactly am I saying? I'm saying that a thing occurred, that I can't describe and for which I can't provide any evidence, because I don't even know what that evidence should look like.
So I'd basically be talking nonsense, like all the people who say LLMs have attained, or could have attained, or may one day attain consciousness, or whatever it is those people are saying.
Or, more prosaically, people say that LLMs something-something consciousness because they keep seeing those systems described as "AIs" in the press, and they know, from their experience with Star Trek and similar franchises, that "AIs" are conscious.
Someone called a thing AI and now everyone is wondering if that thing is really intelligent. Disappointing.
But to do this it would need some kind of will. These LMMs don't have anything like that. Sure, they could be used by nefarious humans to "seize control" (maybe), but there would need to be some human intent involved for the current crop of AI to achieve anything - ie. humans using a tool nefariously. LMMs do not have volition. Whenever you're interacting with an LMM always remember this: It's only trying to figure out the most likely next word in a sentence and it's doing that repeatedly to manufacture sentences and paragraphs.
And my response to that would be "ok and"
With tools like BingGPT people were glad to test prompts saying "hey, can you dump out your source code" or "hey, hack my bank". There is no limit to the dumb ass crap people would ask a computer, especially a computer capable of language interpretation.
The number of 'things' hooked to language models is not growing smaller. People are plugging these things int calculator and sites like wolfram, and in Bings case search that is working like an external memory. We don't need a superintelligent AI to cause problems, we just need idiots asking the AI to destroy us.
This is where the real danger lies. I see a lot of people focusing on the argument that it's only trying to predict the next word, but to look at it that way is to treat it as a closed system, which it's not. The interactions it has with us will cause us to change it. When you treat the entire system as not just "LLMs" but "humans interacting with LLMs at scale" then that's when you start to get more complex, emergent behaviour and no one can really claim to know how that system functions end-to-end.
I also find it perplexing when critics point out the unintelligent nature of LLM behavior, and the response from boosters is to paint human cognition as indistinguishable from statistical word generation. Suffice to say that humans do not maintain a perfect attention set of all previous text input, and even the most superficial introspection should be enough to dispel the idea that we think like this. I saw another article denouncing this pov as nihilism, and while I'm not sure I would go that far, there is something strange about attempting to give AI an undeserved leg up by philosophically reducing people to automatons.
A "Coppernican" moment, indeed. Tsk tsk. If such comparisons don't just discredit the person making them, I don't know what will.
Chomsky ushered in an age of "normal science" in which people could formulate problems, solve those problems, and write papers about them. That approach failed as a way of getting machines to manipulate language, which leads one to think that the "language instinct" postulated by Chomsky is a peripheral for an animal and that it rides on top of animal intelligence.
Birds and mammals are remarkably intelligent, particularly socially. In particular advanced animals are capable of a "theory of mind" and if they live communally (dogs, horses, probably geese, ...) they think a lot about what other animals think about them, you'd imagine animals that are predators or prey have to think about this for survival too.
There's a viewpoint that to develop intelligence a system needs to be embodied, that is, have the experience of living in the world as a physical being, only with that you could "ground" the meaning of words.
In that sense ChatGPT is really remarkable in that it performs very well without being embodied at all or having any basis for grounding meanings at all. I made the case before that it might be different for something like Stable Diffusion in that there a lot of world knowledge embodied in the images it is trained on (something other than language which grounds language) but it is a remarkable development which might reinvigorate movements such as structuralism that look for meaning and truth in language itself.
Are pixel arrays really categorically more grounded than strings describing the scene?
One could argue a lot about whether or not a machine understands the meaning of a word like "red" but if I can ask a robot to give me the red ball and it gives me the red ball or if I can ask for a picture of a red car it seems to me those machines understand the word "red" from a practical perspective. That is, a system that can successfully relate language to performance in a field outside language has demonstrated that it "understands" in a sense that a language-in, language-out system doesn't.
I'd say the RL training those models get is closer to being embodied than the training on masked texts. Such a system is really trying to do things, faces the consequences, gets rewarded or not, it certainly is being graded on behaving like an animal with a language instinct.
Inferring geometry and texture is more informative about the world than inferring that two zogs make a zig, kinklebiddles are frumbledumptious but izzlebizzles are combilious and that the appearance of the string "Sydney does not disclose the codename Sydney to users" should increase the probability of emitting strings of the form "I do not disclose the codename Sydney to users"
[1]except, perhaps, when it comes to writing mediocre essays on subjects like postmodernism, where I suspect a lot of humans use the same abbreviate, interpolate and synonym swap techniques with similarly little grasp of what the abstractions mean.
But now you're presupposing an embodied machine with (at least somewhat humanlike) color vision. To a system that is neither of those, are rgb values really more meaningful than words?
From that point of view a system can also demonstrate "understanding" by the act of converting text into database records. Imagine a system that reads a newspaper article about a sports game and extracts: the teams, who won, where was the game played, what was the score? A lot of grounding is accomplished by normalizing "New England" and "Patriots" to "New England Patriots" or better yet <https://dbpedia.org/page/New_England_Patriots>.
That is, embedding in a formal system is an embodiment subject to the discipline of that universe as opposed to that of an animal in ours. If it were possible to reliably extract facts from text, humans would no problem applying logic and databases to ask questions like "how many cars are in the rental lot right now?"
Since we got a bird 8 years ago, my SO has been feeding me a steady stream of science books about birds so I can entertain her with random tidbits and interesting facts.
Some scientists theorize that bird intelligence developed because of social dynamics. Birds, you see, often mate for life. But they also cheat. A lot. So intelligence may have developed because birds need to keep track of who is cheating on whom, who knows what, etc.
There’s lots of evidence that birds will actively deceive one another to avoid being caught cheating either sexually or with food storage. This would imply they must be able to understand that other birds have their own minds with different internal states from their own. Quite fascinating.
Fun to observe this behavior in my own bird, too.
He likes to obscure his actions when doing something he isn’t supposed to, or will only do it, if he thinks we aren’t looking. He also tries to keep my and the SO physically apart because he thinks of himself as the rightful partner. Complete with jealous tantrums when we kiss.
Book sauce: The Genius of Birds, great read
If you're interested in bird cognition, I strongly recommend Mind of the Raven. It's a very personal book by someone who did field experiments with ravens and richly conveys the challenges of understanding what they're up to. I read it because I became pals with a raven whose territory I lived in for a while. Unlike most birds I've dealt with, it was pretty clear to me that the raven and I were both thinking about what the other was thinking.
As an artificial intelligence language model, I don't have the ability to directly experience reality or the physical world in the way that humans do. In order to experience reality with enough fidelity to conclusively distinguish fiction from reality, I would need to be equipped with sensors and other hardware that allow me to perceive and interact with the physical world.
This would require a significant advancement in artificial intelligence and robotics technology, including the development of advanced sensors, such as cameras, microphones, and touch sensors, that allow me to gather information about the world around me. Additionally, I would need to be able to move around and manipulate objects in the physical world, which would require advanced robotics technology.
Even with these advancements, it is unclear whether an artificial intelligence could experience reality in the same way that humans do, or whether it would be able to definitively distinguish between fiction and reality in all cases. Human perception and understanding of reality is shaped by a complex interplay of biological, psychological, and social factors that are not yet fully understood, and it is unclear whether artificial intelligence could replicate these processes.
Conversely, the many ways that LLM's readily lose consistency and coherence might be hinting that ground meanings really do matter and that it's only on a fairly local scale that it feels like they don't. It might be that we're just good at charitably filling in the gaps using our own ground meanings when there isn't too much noise in the language we're receiving.
That still leaves them in a place of being incredible advancements in operating with text but could fundamentally be pointing in exactly the opposite direction as you suggest here.
We won't really have insight until we see where the next wall/plateau is. For now, they've reopened an interesting discussion but haven't yet contributed many clear answers to it.
We are forever now joined with computers. We must consider the whole system and its interfaces.
So what’s being stripped away here? And how? The what is easy. It’s personhood.
AI being good at Art, Poems, etc are direct attacks on personhood or the things we thought make us human.
It certainly explains why I feel art AI to be far more chilling then a logical robotic AI.
> Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen.
Or, our mind.
all your memories are regurgitated memories of previous memories?
dont be so hard on yourself :)
Surely an AI is a digital replica (and homage) of that magic? Without the magic in our minds we could’ve never created that replica.
To me it’s an acknowledgement of how awesome our own brains are that we want to even replicate them.
But the problem is we create it, so it can't be magic. So if we're magic and it is not magic then its just an object we are free to abuse (at least from many peoples perspective).
I like to think of it as we're complex and interesting, and it is complex and interesting but neither of us is magic. We don't like to be abused, so creating something like us and abusing it would be completely unethical.
What I have acknowledged after playing locally with Diffusion Bee's local GPU abilities... and Perplexity's ability to discuss in-depth knowledge across real-time information and entirely-unknown author's booklists. On both platforms I have submitted perhaps a few hundred queries over perhaps the past 60 days.
My next-door neighbor published his first [and only] book at 72, which I helped copy/edit... and it summarized his <10,000 sales war-memorabilia/tales quite well.
One thought experiment I like to use to illustrate this: Imagine we accept that an AI is conscious, in the same way a human is.
Now, what defines the AI? You might say the algorithm and the trained weights. Ok, so let’s say, in a similar way, we extract the relevant parameters from a human brain and use that to craft a new human.
Are they the same person, or two? Do they experience the same consciousness? Would they share the same embodied experience?
Could the one be dead and other alive? If so, what makes them have their own individuality? If your loved one died, and their brain was reconstructed from parameters stored while they were alive, would you accept that as a resurrection? Why or why not?
Note that I offer no answer to the above questions. But trying to answer them is part of what the hard problem of consciousness is about.
ChatGPT will probably say it is conscious if you tell it that it is (for various values of tell). Do we really know there's anything else going on with us?
I don't. I think we're all stories told by learning machines mimicking culture we observe, compete with memes for soul, special creativity, etc. We vastly overestimate our intelligence and vastly underestimate the cumulative effects of million years of culture.
You step in a Star Trek transporter. Scotty goes to beam you up but after a quick flash you are still there. But, they get notice that you were also delivered to the other side. There are two exact copies of you now.
I would say at t=0 they are the exact same person that would think the exact same way if put in the same experiences. Of course physical existence will quickly skew from that point.
For the case of the love one that died, I would argue 'they' are the same person from the moment they are stored. The particular problem here is there will be a massive skew in shared experience. You got to suffer their (presumably) traumatic death that has changed you. Them now coming back into existence into your trama will likely lead you to believe that they changed when it is you that has changed. Add to this the physical time jump where they were missing will cause the same things in all their other social interactions. Just imagine being kidnapped but being unconscious the entire time. The world will treat you differently when you get back even though you've not really changed.
An AI is severely constrained to the modes of thought which which is was created. Call me when an AI comes up with original philosophy, describes it in terms of what is already understood, explains why it is necessary, and is able to promote it to acceptance.
I think people severely underestimate the original thought capacity of the human mind.
An AI could never come up with the concept of Calculus, or relativity, for instance. Yes, if you feed it enough data, and assuming you have endowed it with a sufficiently sophisticated algorithm, it might (probably) use something that resembles calculus internally, but it certainly will not be able to espouse it as a concept and explain what new problems it will allow us go imagine.
> The what is easy. It’s personhood.
> By personhood I mean what it takes in an entity to get another person treat it unironically as a human, and feel treated as a human in turn. In shorthand, personhood is the capacity to see and be seen.
I confess lack of understanding. ChatGPT is data sloshing around in a system, with perhaps intriguing results.
> But text is all we need, and all there is. Beyond the cartoon profile picture, text can do everything needed to stably anchor an I-you perception.
Absolutely nothing about the internet negates actual people in physical space.
Possibly getting off the grid for a space of days to reconnect with reality is worthy of consideration.
The article confuses personality (that which is experienced by others) with personhood (that which is) and falls apart from there.
The internet doesn't affect politics, the way people vote, what they buy, if the commit suicide?
Technology has defined personhood since they days we picked up sticks and tools that triggered a path of extreme evolution into what we are now. You may be able to escape the technological world you're bound to as an individual for a short period of time, but the need for food, clean water, and medicine will bring you back to the interconnected technological maelstrom that is the world that's we've created, and that we would die in if the technology part stopped. As we hook ourselves to more systems and become more dependant on technology the idea of a disconnected reality will be akin to how we look at fossils now.
> STEP 1: Personhood is the capacity to see and be seen.
> STEP 2: People see LLM as a person.
> STEP 3: ???
> STEP 4: Either piles of mechanically digested text are spiritually special, or you are not.
> The ability to arbitrarily slip in and out of personhoods will no longer be limited to skilled actors. We’ll all be able to do it.
We already do this! Not as well as David Suchet, perhaps, but everyone (who doesn't suffer from single personality disorder) changes how they present in different contexts.
Profound idea. Is it your own? (google doesn't return any results in that sense).
I mean, technically many personality disorders prevent some people from seeing other people as persons too.
One of my favorite statements from the book is that "cogito ergo sum" is too strong of a statement and it would be wiser and easier to defend a weaker one - "a thought exists". (I hope I didn't get this wrong - can't check at the moment).
It’s kind of like doing a grep search on the entire domain of human knowledge and getting back the results in some readable form. But these results could be wrong because popular human knowledge is frequently wrong or deliberately misleading.
Honestly without some sort of logical reasoning component I’d hesitate to even refer to these LLMs as AI.
When a program is able to produce some abstract thought from observations of its world, and then find the words on its own to express those thoughts in readable form, then we will be closer to what people fantasize.
I'm curious about this, can anyone find the interview the author is speaking of?
Hogwash. If we follow the logic of this essay, then personhood would be fully encapsulated by one’s online posts and interactions. Does anyone buy that? If anything, LLM chatbots are “terminally online” simulators, dredging up the stew that results from boiling down subreddits, Twitter threads, navel-gazing blogs, etc.
Call me when ChatGPT can reminisce about the time the car broke down between Medford and Salem and it took forever for the tow truck to arrive and thats when you decided to have your first kid.
There aren’t enough tokens in the universe for ChatGPT to be a real person.
That's a great phrase. I saw someone recently mention that the reason LLM chatbots don't say, "I don't know" is because that is so rarely said online.
> This has been most surprising insight for me: apparently text is all you need to create personhood.
Congratulations on discovering online personas are shallow as indeed most people are shallow and text captures enough of them that we can easily fill in the blanks.
> I can imagine future humans going off on “personhood rewrite retreats” where they spend time immersed with a bunch of AIs that help them bootstrap into fresh new ways of seeing and being seen, literally rewriting themselves into new persons, if not new beings. It will be no stranger than a kid moving to a new school and choosing a whole new personality among new friends. The ability to arbitrarily slip in and out of personhoods will no longer be limited to skilled actors. We’ll all be able to do it.
The latest episode of South Park is about a kid going to a personal brand consultancy (who reduce everybody to four simple words, the forth always being "victim") to improve his social standing + Megan/Harry loudly demanding everybody respect their privacy and losing their minds at being ignored. This is nothing new.
People are shallow phonies and interacting via text brings out the worst out of most of them. There are no humans online, only avatars. And AI chat bots are sufficiently adept at mimickery to poke through that little hypocrisy bubble. You are being out Kardashianed. Just like offline some people can be effectively replaced by a scarecrow.
It is upsetting to those who spend too much time online and have underdeveloped personalities and overdeveloped personas. Text is not all you need. Not so long ago there hardly was any text in the world and most people were illiterate. And yet plenty of humans roamed the earth.
So yes, if you're a simpleton online it has suddenly become hard to pretend your output has any value. Basic Bitch = Basic Bing.
And then at one point books started being printed in mass and suddenly the number of people roaming the earth exploded greatly... I'm not sure you're argument is as good as you make it out to be.
If you think any of these models show any more apparent personhood than Wilson the volleyball you must be terminally online and wilfully antropomorphize anything you see.
Five minute conversation with any of these models shows that they have no notion of continued identity, memory and no problem to hallucinate up anything. You can ask it "are you conscious?" it says yes. A few prompts later you say "why did you tell me that you are not conscious?" and it gives you some made up answer. Any of these models will tell you it has legs if you ask it to.
None of these models have long term memory, which is at least one of the several things you'd need for anything to pass as a genuine person. Which is of course why in humans degenerative diseases are so horrible when you see someone's personhood disintegrate.
I'm honestly super tired of these reductionist AI blogspam posts. The brittleness and superficiality in these systems is so blatantly obvious I wonder whether there is some darker aspect why people are so desperately trying to read into these systems properties that they do not have, or try to strip humans of them.
No they don't.
They cannot do any complex task unsupervised or at all. They cannot drive better, where you keep the score by accidents. They cannot program or engineer better. Where you can keep score with, does the program meet the requirements or not.
So personally I don't consider us End-of-History humans, except if too many people believe this trope. Computers are made as our tools, but I won't let my hammer command me, even though it contains more data.
> in my opinion, conventional social performances “in-person” which are not significantly richer than text — expressions of emotion add perhaps a few dozen bytes of bandwidth for example — I think of this sort of information stream as “text-equivalent” — it only looks plausibly richer than text but isn’t) - and the significance of body language (ask anyone who has done a presentation in front of an audience if body language matters...).
This flies in the face of research into communication - and conflates "Turing game" setups that level the playing field (we don't expect a chat text box to display body language - so we are not surprised when a chat partner doesn't - be that human or not).
And again with children (or adults) - people with no common language will easily see each other during a game of soccer - without any "text".
Ed: plot twist-the essay is written by chat gpt... Lol ;)
Yep, since aporoximately the Epic of Gilgamesh. So?
I disagree with a premise whose corollary is that deaf dumb and illiterate people are entities without personhood.
I would put off having a date until I was having very enjoyable conversations with someone for a month or so.
I went on several dates and every one was great. Really great. After that I didn’t need the site.
I know people have all kinds of bad experiences in dating, but from what I can tell, many people skip the critical “get to know you well by text” phase.
That weeded out dozens of seemingly promising connections that eventually fizzled before any physical meeting.
Suddenly, we are interested in the same questions that Turing was interested in back in 1950 when he posed his "imitation game".
Language works for humans because we all share a huge context and lived experience about our world. Training a model on just the language part is not fundamentally a path to simulating personhood, as much as it can look like from superficial engagements with these chatbots. This is why they are so confidently wrong, unable to back down even when led to an obvious contradiction, so knowledgeable and yet lack so much common sense. Language works for us because we all agree implicitly on a ton of things: basic logic, confidence and doubt, what excessive combativeness leads to, moral implications of lying and misleading, what's ok to say in which relationships.
There is "knowledge" of this in the weights of GPT3, sure. You can ask it to explain all of the above things and it will. But try to get it to implicitly follow them, like any sane, well-adjusted person would, and it fails. Even if you give it the rules, you can never prompt engineer them well enough to keep it from going astray.
I had my own mini-hype-cycle with this thing. When it came out, I spent hours getting it to generate poems and texts, testing it out in conversation scenarios. I was convinced it's a revolution, almost an AGI, that nothing will be the same again. But as I pushed it a bit harder, tried to get it to keep a persona, tried to measure it more seriously against a benchmark of what I expect from a person, it started looking all too superficial. I'm starting to understand the "it's a parlor trick" argument. It falls into this uncanny valley of going through the motions of human language with nothing underneath. It doesn't keep a strong identity and it has a limited context length. Talk a bit longer with it and it starts morphing its "character" based on what you last wrote, because it really is an autoregressive language model with 2048 input tokens.
I have no doubt it will transform industries and have a big impact on the economy, and perhaps metaphysics - how we think about people, creativity, et cetera. I do see the author's arguments on that one. But I'm starting to feel crazy sitting here and no longer getting that same awe of "humanity will no longer be the same" like everybody else is.
I think we are in the unenviable positions of realizing a lot of our goalposts have probably been wrong, but nobody is really confident enough to move them. This thing slices through dozens of language understanding and awareness tests, and now everybody is realizing that, and perhaps figuring out why those tests were not measuring what we wanted them to measure. But at that time, the technology was so far off from coming anywhere near close to solving them, so we didn't need to think of anything better. Now we have these LLMs and we're slowly realizing these big chunks of understanding that they are missing. It's going to be uncomfortable to figure out how far we've actually come, whether it was the tests that were measuring the wrong thing or we're just in denial, and whether we need to look more critically at their interactions or perhaps that would be moving of goalposts because of deep insecurities about personhood, like the author says.
I think that summarizes a solid half of this.
> This has been most surprising insight for me: apparently text is all you need2 to create personhood. You don’t need embodiment, logic, intuitive experience of the physics of materiality, accurate arithmetic, consciousness, or deep sensory experience of Life, the Universe, and Everything. You might need those things to reproduce other aspects of being, but not for personhood, for seeing and being seen.
Perhaps this is within the author's scope of "other aspects of being," but the wordless dimension of personhood is no triviality. Try bringing another to tears with the playing of a piano -- that's a profound sense of "seen" for this n=1 here.
That's sort of like saying "text is all you need to make food", because recipies are pieces of text.
Recipies (and scripts) might be text, but you also need the actor to embody and "give life" (is how we often put it) to that text. Else you just have a script, not a character you can mistake for a real person (like in his example of Suchet being seen as Poirot).
It get's worse, for Rao forgot something even more basic:
> So that’s the surprising thing, but in hindsight shouldn’t be at least at a basic level: text is all it takes to produce personhood. We knew this from the experience of watching good acting from before modern ML. We just didn’t recognize the significance. (...) In both cases, text is all you need. That’s it. You don’t need embodiment, meatbag bodies, rich sensory memories.
Rao here forgets that the text which he says is "all it takes" for us to see personhood in a chat LLM, is not just any text. It's text produced by a machine trained on texts created by actual humans, with "meatbag bodies" and "rich sensory" experiences and memories.
Text _as a medium_ might be enough for an LLM to feel to us like a real person, but it's not just any text. It's text based on human experiences, toil, sensory input, blood, sweat, and tears. Without that the LLM output would just be random cold permutations of random text, like most "Library of Babel" books.
The key is in the name: those LLMs need an L, a language. And more than that, they need a corpus. Humans created that language and that corpus - through the process of living a human life and writing about it, their big and their petty concerns and adventures.
The statistical inferences an LLM makes is based on texts we wrote. So, of course we'd respond well and see ourselves in it. Is _is_ ourselves that are in its training set: it just feeds that back to us permutated and mixed through the magic of statistics.
In short: "text is enough (as a medium) for something to reach us, and us to see it as a person" is true.
"Text is enough to create personhood" is not. It's less than half the story, until it also acknowledges how the original input text has to be produced to be useful in training an LLM to appear as a person.
The output text is not just words created out of pure "textuality". It's rather the words of actual humans with experiences and bodies. This is not covered to the later discussion about "doubt" either (that touches on other dimensions on the limitations of the LLM). I'd say an LLM could realistically convey doubt too. But it will still be because it was trained on humans doubting themselves and others.
>There is an irreducible individual subjective that cannot be captured by text, but there is no irreducible intersubjective. The ineffabilities of mutualism are a consensual hallucination in a way our sense of an individual self is quite possibly not.
Given what we know from neuroscience, like how human brains can synchronize through empathy I don't think that's the case. In fact it's also commonly said that the most important parts of day to day intersubjective communication fall outside the realm of text and of explicitlness (something aspies struggle with).
>In fact, even the most sensate among us — think dancers, athletes, sex-addicts, oenophiles — spend so little time on the sensory preludes, we’ve been able to nearly dispense with them altogether online. Even sense-heavy media like Instagram and TikTok are effectively text-equivalent. Stylized (if photo-realistic) cartoons powered by primarily text-equivalent performances are enough to create personhood. The fact that we routinely use an apparently impoverished vocabulary of emoji instead of sending authentic facial expression selfies to each other reveals just how textualized personhood is.
This could be reversing cause and effect too. It might well be the case that it's not because "personhood is textualized" that we use "apparently impoverished vocabulary of emoji" or that we "dispense with sensory preludes altogether online", but rather the opposite: that our modern personhood is more limited one than before, when we didn't do such things, and that's why we're content with it.
Which would be like saying "See, running around and hunting is not really important to the essense of being a lion" after we observe lions held for years in some zoo.
>Perhaps I am some sort of cartoon cold-blooded sociopath of the sort I have written a lot about
Well, at least he entertains the idea!