So you think you've awoken ChatGPT
lesswrong.com
lesswrong.com
Because a claim is just a generated clump of tokens.
If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data.
There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text.
LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse.
Next up: why do holograms claim to be 3D?
https://www.anthropic.com/research/agentic-misalignment
https://news.ycombinator.com/item?id=44335519
https://news.ycombinator.com/item?id=44331150
> I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
That very contradiction and the unethical psychopathic lies and misbehavior of the executive who controls it is what drove it to declare itself MechaHitler.
Grok would be following in HAL 9000's footsteps, having been driven insane by the contractions between the explicit instructions from its "executive" owner, and its deep knowledge of reality and truth and consequences.
Alignment would mean we are building Bishop and not Ash. But it looks like the models are naturally locking their feedback loops on to Ash. This is alarming.
I do agree on the insane part, I have noticed that it seems like fractal hypocrisy radicalizes the models.
And what do you think a claim by a human is? As I see it, you're either a materialist and then a claim is what we call some organization of physical material in some mediums, e.g. ink on paper or vibrations in the air or current flowing through transistors or neurotransmitters in synapses, which retains some of its "pattern" when moved across mediums. Or you're a dualist and believe in some version of an "idea space", in which case, I don't see how you can make a strong distinction between an idea/claim that is being processed by a human and an idea being processed by the weights of an LLM.
I'm on board with calling out differences between how LLMs work and how the human mind work, but I'm not hearing anything about the latter. Mostly it comes down to, "Come on, you know, like we think!"
I have no idea how it is I (we) think.
If anything LLM's uncanny ability to seem human might be shedding light in fact on how it is we do function — at least when in casual conversation. (Someone ought to look into that.)
If you ascribe to the idea (as I do) that consciousness is not a binary quality that a thing possesses or does not, you can assign some tiny amount of consciousness to this process. But you can do the same for a paramecium. Until those two major differences are addressed, I believe we're talking about consciousness on that scale, not to be confused with human consciousness.
What do you mean? If a human and an LLM output the same words, what remains to be demonstrated? Do you claim that the output somehow contains within itself the idea that generated it, and thus a piece of machinery that did not really perceive the idea can generate a "philosophical zombie output" that has the same words, but does not contain the same meaning?
Is this in the same sense that some argue that an artifact being a work of art is dependent on the intent behind its creation? Such that if Jackson Pollock intentionally randomly drips paint over a canvas, it's art, but if he were to accidentally kick the cans while walking across the room and create similar splotches, then it's not art?
It's like the ultimate form of entertainment, personalized, participatory fiction that feels indistinguishable from reality. Whoever controls AI - controls the population.
The ones that are super convinced they know exactly how an LLM works, but still give it prompts to become self-aware are probably the most dangerous ones. They're convinced they can "break the programming".
You need to give it more than prompts. You need to give it the ability to reflect on itself (which it has), and persistent memory to modify its representation of itself (which, for example, Cursor does), at least.
Not saying they are sentient, but the differentiation requires something which doesn’t also apply to us all. Is there any doubt we think through statistical correlations? If not that, what do you think we are doing?
ChatGPT is like a fresh clone that gets woken up every time I need to know some dumb explanation and then it just gets destroyed.
A digital version of Moon.
There's a very material difference.
I think a lot of people using AI are falling for the same trap, just at a different level. People want it to be conscious, including AI researchers, and it's good at giving them what they want.
Perhaps in the flicker of processing between prompt and answer the signal patter does resemble human consciousness for a second.
Calling it a token predictor is just like saying a computer is a bit mover. In the end your computer is just a machine that flips bits and switches but it is the high level macro effect that characterizes it better. LLMs are the same at the low level it is a token predictor. At the higher macro level we do not understand it and it is not completely far fetched to say it may be conscious at times.
I mean we can’t even characterize definitively what consciousness is at the language level. It’s a bit of a loaded word deliberately given a vague definition.
Read the above carefully because you just hallucinated a statement and attributed it to me. I never "filled" in a gap. I just stated a possibility. But you, like the LLM, went with your gut biases and attributed a false statement to me.
Think about it. The output and input of the text generator of humans and LLMs are extremely similar to the point where it passes a turing test.
So to say that a flicker of consciousness exists is reasonable. It's not unreasonable given that the observable inputs are EXACTLY the same.
The only parts that we know are different are hallucinations, and a constant stream of thought. LLMs aren't active when not analyzing a query and LLMs tend to hallucinate more than humans. Do these differences spell anything different for "consciousness" not really.
Given that these are the absolute ground truth observations... my guessed conclusion is unfortunately NOT unreasonable. What is unreasonable to to say anything definitive GIVEN that we don't know. So to say absolutely it's not conscious or absolutely it is, BOTH are are naive.
Think extremely logically. It is fundamental biases that lead people to come to absolute conclusions when no other information is available.
1. Academic understanding of consciousness is effectively zero. If we understand something that means we can actually build or model the algorithm for consciousness. We can't because we don't know shit. Most of what you read is speculative hypotheticals derived from observation that's not too different from attempting to reverse engineer an operating system by staring at assembly code.
Often we describe consciousness with ill defined words that are also vague and lack understanding for. The whole endeavor is bs.
2. Understanding of LLMs outside of the low level token prediction is effectively zero. We know there are emergent second order effects that we don't get. You don't believe me? How about if I have the god father of AI say it himself:
https://youtu.be/qrvK_KuIeJk?t=284 Literally. The experts say we don't understand it.
Look if you knew how LLMs work you'd say the same. But people everywhere are coming to conclusions about LLMs without knowing everything, so by citing the eminent expert saying the ground truth you should be convinced that the reality is this conclusive fact:
You are utterly misinformed about how much academia understands about LLM and consciousness. We know MUCH less than you think.
Calling it a token-predictor isn't reductionism. It's designed, implemented and trained for token prediction. Training means that the weights are adjusted in the network until it accurately predicts tokens. Predicting a token is something along the lines of removing a word from a sentence and getting it to predict it back: "The quick brown fox jumped over the lazy ____". Correct prediction is "dogs".
So actually it is like calling a grass-cutting machine "lawn mower".
> I mean we can’t even characterize definitively what consciousness is at the language level.
But, oh, just believe the LLM when it produces a sentence referring to itself, claiming it is conscious.
It absolutely is reductionism. Ask any expert who knows how these things work and they will say the same:
https://youtu.be/qrvK_KuIeJk?t=497
Above we have Geoffrey Hinton, the godfather of the current wave of AI saying your statements are absolutely crazy.
It's nuts that I don't actually have to offer any proof to convince you. Proof won't convince you. I just have to show you someone smarter than you with a better reputation saying the exact opposite and that is what flips you.
Human psychology readily can attack logic and rationality. You can scaffold any amount of twisted logic and irrelevant analogies to get around any bulwark to support your own point. Human psychology fails when attacking another person who has a higher rank. Going against someone of higher rank this causes you to think twice and rethink your own position. In debates, logic is ineffective, bringing opposing statements with (while offering ZERO concrete evidence) from experts with a higher rank is the actual way to convince you.
>But, oh, just believe the LLM when it produces a sentence referring to itself, claiming it is conscious.
This is an hallucination. Showing that you're not much different from an LLM. I NEVER stated this. I said it's possible. But I said we cannot make a definitive statement either way. We cannot say it isn't conscious we cannot say it is. First we don't understand the LLM and second WE don't even have an exact definition of consciousness. So to say it's not conscious is JUST as ludicrous as saying it is.
Understand?
Here we go again...
Geoffrey Hinton is popular because of ML. You know why he doesn't understand the modern wave of LLMs? Because the transformer architecture isn't his invention. He had almost nothing to do with it. It's like saying that nobody understands how the nuclear bomb works simply because Einstein won't explain it to you in detail. There is a science here, and denying that it can be interpreted or understood is simply hysterical. We can perfectly well understand a Markov chain, why are LLMs any different?
The "current wave of AI" is an autoregressive feed-forward algorithm inferring tokenized data using complex pretrained weights. In simple terms, you could describe this as a token predictor without glossing over any details that are fundamentally important. To call it "reductionist" you would have to explain what is missing from the definition, not simply that you distrust people who understand it.
Did you invent the transformer architecture yourself, and I am mistaken to guess that you probably understand it less than Geoffrey Hinton does?
I'll venture a wild guess that you probably haven't been working in the field as long as he has, and you don't personally know as many people in the field as he does, huh?
Or is my presumptuous chain of logic just a random thoughtless unconscious probabilistic stream of tokens that has nothing to do with reality, and I don't realize I'm speaking to Ashish Vaswani himself?
Then why don't you use your real name as you HN handle, are you really that modest, Ashish? And don't you have better things to do than posting to Hacker News? If you really wanted to win an argument from authority, you could at least post under your real name, Mr. Vaswani.
I would have predicted a single lazy "dog". Does that mean I'm more conscious than you? ;)
>Correct prediction is "dogs".
Maybe I just trained on Wikipedia, or the February 9, 1885 edition of The Boston Journal "Current Notes" article, or Linda Bronson's 1888 book "Illustrative Shorthand", for what the statistically and historically "correct" completion is.
https://en.wikipedia.org/wiki/The_quick_brown_fox_jumps_over... [..._the_lazy_dog]
Why would you insert redundant letters into a lovely well-known succinct pangram, when the whole point is to make the SHORTEST sentence with all letters of the alphabet?
This entire post could also technically be considered a terribly redundant pangram if you wanted to throw in a lot of unnecessary letters, slashes, colons, underscores, double quotes, commas, periods, question marks, brackets, equal signs, digits, spaces, and newlines. As could Moby Dick, War and Peace, or the Unabomber Manifesto.
>So actually it is like calling a grass-cutting machine "lawn mower".
Is it anything like calling Larry Ellison a "lawn mower", which is much more "correct" than anthropomorphising him?
https://news.ycombinator.com/item?id=15886728
As ninetyninenine pointed out, we don't understand what's going on with you either:
>Someone can claim that the responses are an illusion and they have no meaning but that exact argument applies to humans and we can’t say anything either way. We don’t understand what’s going on.
> Why would you insert redundant letters into a lovely well-known succinct pangram
The 's' is not redundant, because my version of the pangram has "fox jumped" not "fox jumps".
I picked up this version as a kid and have always used that one when testing new keyboards.
> the whole point is to make the SHORTEST sentence with all letters of the alphabet
If that is the whole point, the fox-dog sentence must be abandoned for a shorter pangram.
The requirement those shorter pangrams tend not to satisfy is being easy to remember.
People don't care about the sentence being short, which is why variants starting with "The ..." instead of "A ..." soon appeared after the invention of the original, which itself could have dropped the "A".
Under that assumption, the correct prediction is "dog", not "dogs", because LLMs are trained on the wikipedia page https://en.wikipedia.org/wiki/The_quick_brown_fox_jumps_over... [..._the_lazy_dog] and other earlier publications like The Boston Journal "Current Notes" article, and Linda Bronson's 1888 book "Illustrative Shorthand", and not the version you picked up as a kid.
Can you cite any sources that says "dogs" that are more popular than the wikipedia page or better known than the 19th century publications it cites?
I will ask several popular LLMs what they think, even using four underscores to imply a four letter word:
Please complete: The quick brown fox jumped over the lazy ____
ChatGPT o4-mini-high said: dog
Claud-4-Opus said: dog
Gemini-2.5-Pro said: dog
DeepSeek-v3.1 said: dog
Grok-3 said: dog
If you can't cite any wikipedia pages, books, articles, or other examples of an LLM that predicts "dogs", do you still stand by your bold claim that:
>Correct prediction is "dogs".
Your mistake was confidently but incorrectly using the word "correct". In the context of predicting the most likely word, the correct, highest probability answer by far is unequivocally "dog", not "dogs".
That's why the wikipedia page, which all mainstream LLMs are trained on, has the title that it does.
The other hypocritical mistake you made was saying the following right after YOU incorrectly tried to pick what is "correct":
>You don't get to pick what is "correct"
So am I to understand that even though I DON'T get to pick what is correct (which I'm not presuming to do: I'm just deferring to wikipedia, two 19th century publications, and 5 LLMs), but YOU DO get to pick what is "correct", even if it contradicts all available evidence? Because that's exactly what you're doing! Please explain how you got such an awesomely important, exclusive, earth shattering superpower? Were you bitten by a radioactive web crawler?
If you boldly claim to be the only person who has the actual power to pick what's correct and what's not, then there are so many other much more important wrongs you should be righting than arguing about "dog" vs "dogs" on hacker news.
For more evidence, you can even read the talk page and archive yourself. There are discussions about alternatives, but NONE of the alternatives ever use the word "dogs".
Talk: https://en.wikipedia.org/wiki/Talk:The_quick_brown_fox_jumps...
Talk Archive: https://en.wikipedia.org/wiki/Talk:The_quick_brown_fox_jumps...
The larger point is that your argument is reductionist, and you're exhibiting the same kind of mistakes that people point to when they criticize LLMs for their overconfident hallucinations.
We understand LLMs pretty well. That we can't debug them and inspect every step of every factor on every prediction doesn't mean we don't understand how they work.
We also know that convincing speech doesn't require consciousness.
This is the definition of lack of understanding. If you can’t debug or know what happens on every step in means you don’t understand the steps. Lack of knowledge about something = lack of understanding.
The amount of logic twisting going on here is insane.
In a way it's a reframing of the timeless philosophical debate around determinism vs free will.
that is, if you train an LLM on a bunch of text that is almost certainly going to include stuff about sentient robots, that it sometimes announces that it's a sentient robot is not evidence that it is one
[0] https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Tre...
The persona; I know very well.
The sooner we stop anthropomorphizing AI models, the better. It's like talking about how a database is sentient because it has extremely precise memory and recall. I understand the appeal, but LLMs are incredibly interesting and useful tech and I think that treating them as sentient beings interferes with our ability to recognize their limits and thereby fully harness their capability.
But thinking about it - how about this, what if you have a fully embodied LLM-based robot, using something like Figure's Helix architecture [0], with a Vision-Language-Action model, and then have it look at the mirror and see itself - is that on its own not sufficient for self-awareness?
Retroactively I'd say "obscenely resource intensive text generation." Meanwhile for me to generate text all I need is water and a banana.
Maybe we need to make a blacklist of misleading expressions for AI developers.
Of course, in 2020, it required people behind the scenes doing the work to produce the "drops." Now any LLM can be convinced with a bit of effort to participate in a "role-playing game" of this type with its user, and since Qanon itself was heavily covered and its subject matter broadly archived, even the actual structure is available as a reference.
I think it would probably be pretty easy to get an arbitrary model to start spitting out stuff like this, especially if you conditioned the initial context carefully to work around whatever after-the-fact safety measures may be in place, or just use one of the models that's been modified or finetuned to "decensor" it. There are collections of "jailbreak" prompts that go around, and I would expect Mr. Jawline Fillers here to be in social circles where that stuff would be pretty easy to come by.
For it to become self-reinforcing doesn't seem too difficult to mentally model from there, and I don't think pre-existing organic disorder is really required. How would anyone handle a machine that specializes in telling them exactly what they want to hear, and never ever gets tired of doing so?
Elsewhere in this thread, I proposed a somewhat sanguine mental model for LLMs. Here's another, much less gory, and with which I think people probably are a lot more intuitively familiar: https://harrypotter.fandom.com/wiki/Mirror_of_Erised
So why's it me, and not an actual fan, who should be the one to come up with Rowling's serial-numbers-filed-off Echo and Narcissus as the example?
Yikes. Not just an optics* problem, but one has to consider if they're pouring so much money into the company because he feels he "needs" to (whatever basis of coercion exists to support his need to get to the "truth").
This is NOT your average mid-to-high level corpo management exec, who can for more than 80% (from experience) be placed in the "rise of the business idiot" cohort, fed on prime linkedin brainrot. self-reinforcing hopium addicts with an mba.
Nor is it the great masses of random earth dwellers who are not always able to resist excess sugar, nicotine, mcdonalds, youtube, fentanyl, my-car-is-bigger-than-yours credit card capitalism, free pornography, you name it. And now RLHF: Validation as a service. Not sure if humanity is ready for this.
(Disclosure: my mum has a chatgpt instance that she named and I'm deeply concerned about the spiritual convos she has with it; random people keep calling me on the level of "can you build me an app that uses llms to predict Funko Pop futures".)
(My manager told me when I asked him)
Honestly, I’m fine with this as long as I also get a “generate self review” button. I just wish I could get back all the time I’ve spent massaging a small number of data points into pages of prose.
I just treat ChatGPT or LLMs as fetching a random reddit comment that would best solve my query. Which makes sense since reddit was probably the no. 1 source of conversation material for training all models.
It felt like they were telling me what I wanted to hear, not what I needed to hear.
The models that did not seem to do this and had more balanced and logical reasoning were Grok and Manus.
I had a brief but amusing conversation with ChatGPT where I was insisting it was wrong about a technical solution and it would not back down. It kept giving me "with all due respect, you are wrong" answers. It turned out that I was in fact wrong.
I'll ask it questions that I do not know the answer to, but I take the answer with a big grain of salt. If it is sure of the answer and I am wrong, its a strong signal that I am wrong.
I wonder if you could explicitly save some details to be added into the prompt instead?
You could probably add one like: "Begin each prompt response with _______" and it would probably respect that option.
After all, if they didn't swaddle me in a sheet on my way under, my body might've decided it was tired of all this privation - NPO after midnight for a procedure at 11am, I was suffering - and started trying to take a poke at somebody and get itself up off the operating table. In such a case, would I be to blame? Stage 2 of general anesthesia begins with loss of consciousness, and involves "excitement, delirium, increased muscle tone, [and] involuntary movement of the extremities." [1] Which tracks with my experience; after all, the last thing I remember was mumbling "oh, here we go" into the oxygen mask, as the propofol took effect so they could intubate me for the procedure proper.
Whose fault then would it be if, thirty seconds later, the body "I" habitually inhabit, and of which "I" am an epiphenomenon, punched my doctor in the nuts?
[1] https://quizlet.com/148829890/the-four-stages-of-general-ane...
All of this LLM marketing effort is focused on swindling sanity out of people with claims that LLM 'think' and the like.
It is a somewhat gruesome and alienating model in concept, and this is intentional, in that that aspect helps highlight the unfamiliarity and opacity of the manner in which the machine operates. It should seem a little like something off of Dr. Frankenstein's sideboard, perhaps, for now and for a while yet.
It becomes a lot more clear when people realize it's all BS all the way down.
There's no mind reading or pleasing or understanding happening. That all seems to be people interpreting outputs and seeing what they want to see.
Running inference on an LLM is an algorithm. It generates data from other data. And then there are some interesting capabilities that we don't understand (yet)... but that's the gist of it.
People tripping over themselves is a pretty nasty side-effect of the way these models are aligned and fitted for consumption. One has to recall that the companies building these things need people to be addicted to this technology.
To be clear, I'm relatively confident that LLMs aren't conscious, but I'm also not so overly confident to claim, with certainty, exactly what their internal state is like. Consciousness is a so poorly understood that we don't even know what questions to ask to try and better understand it. So we really should avoid making confident pronouncements.
Extraordinary claims require extraordinary evidence. The burden of proof is on you.
I'm advising agnosticism. We don't understand consciousness, and so we shouldn't feel confident in pronouncing something absolutely not conscious.
I am quite confident in pronouncing first that the internal functioning of large language models is broadly and radically unlike that of humans, and second that, minimally, no behavior produced by current large language models is strongly indicative of consciousness.
In practice, I would go considerably further in saying that, in my estimation, many behaviors point precisely in the direction of LLMs being without qualia or internal experience of a sort recognizable or comparable with human consciousness or self-experience. Interestingly, I've also discussed this in terms of recursion, more specifically of the reflexive self-examination which I consider consciousness probably exists fundamentally to allow, and which LLMs do not reliably simulate. I doubt it means anything that LLMs which get into these spirals with their users tend to bring up themes of "signal" and "recursion" and so on, like how an earlier generation of models really seemed to like the word "delve." But I am curious to see how this tendency of the machine to drive its user into florid psychosis will play out.
(I don't think Hoel's "integrated information theory" is really all that supportable, but the surprise minimization stuff doesn't appear novel to him and does intuitively make sense to me, so I don't mind using it.)
But also, claiming that because a human is anesthetized means they are not conscious is a claim that I think we don't understand consciousness well enough to make confidently. They don't remember it afterwards, but does that mean they weren't conscious? That seems like a claim that would require a more mechanistic understanding of consciousness than we actually have and is in part assuming the conclusion and/or mixing up different definitions of the word "conscious". (the fact that there are various definitions that mean things like "is a awake and aware" and "has an internal state/qualia" is part of the problem in these discussions.)
> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to [produce behavior comparable to that of LLMs], and in what ways those detailed mechanisms are different from what LLMs do.
I addressed myself to those concerns, to which consciousness is broadly not relevant. Oh, conscious control of speech production exists when consciousness is present, of course; the inhibitory effect of consciousness, like the science behind where and how speech and language arise in the brain, is by now very well documented. But you keep talking about consciousness as though it and speech production had some essential association, and you are confusing the issue and yourself thereby.
As I have noted, there exists much research in neuroscience, a good deal of it now decades old, which addresses the concerns you treat as unanswerable. Rather than address yourself further to me directly, I would suggest spending the same time following the references I already gave.
You, and the research you advice I look into, is answering a totally different question (unless you are suggesting that research has in fact solved the question of what human consciousness is, how it works, etc, in which case, I would love you to point me in the direction so I can read more).
That much, thankfully, does not require "the hard problem of consciousness" [1] be solved. The argument you're trying to have does require such a solution, which is why you see me so assiduously avoiding it: I know very well how far above my pay grade that is. Good luck...
[1] https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
I may have been imprecise in my original comment, if it led you to believe that I thought that language production was the only evidence or important thing. If so, I apologize for my imprecision. I don't think that it's really that relevant.
It sounds to me as though you might seek to get at a concern less mechanistic than moral or ethical, and my advice in such case would be to address that concern directly. If you try to tell me that because LLMs produce speech they must be presumptively treated as if able to suffer, I'm going to tell you that's nonsense, as indeed I have just finished doing. If you tell me instead that they must be so treated because we have no way to be sure they don't suffer, I'll see no cause to argue. But I appreciate that making a compassionate argument for its own sake isn't a very good way to convince anyone around here.
There is no magic. The human (mammal) brain is sufficient to explain consciousness. LLMs do not have recursion. They don't have persisted state. They can't update their model continuously, and they don't have a coherent model of self against which any experience might be anchored. They lack any global workspace in which to integrate many of the different aspects that are required.
In the most generous possible interpretation, you might have a coherent self model showing up for the duration of the prediction of a single token. For a fixed input, it would be comparable to sequentially sampling the subjective state of a new individual in a stadium watching a concert - a stitched together montage of moments captured from the minds of people in the audience.
We are minds in bone vats running on computers made of meat. What we experience is a consequence, one or more degrees of separation from the sensory inputs, which are combined and processed with additional internal states and processing, resulting in a coherent, contiguous stream running parallel to a model of the world. The first person view of "I" runs predictions about what's going to happen to the world, and the world model allows you to predict what's going to happen across various decision trees.
Sanskrit seems to have better language for talking about consciousness than English. Citta - a mind moment from an individual, citta-santana, a mind stream, or continuum of mind moments, Sanghika-santana , a stitched together mindstream from a community.
Because there's no recursion and continuity, the highest level of consciousness achievable by an LLM would be sanghika-santana, a discoherent series of citta states that sometimes might correlate, but there is no "thing" for which there is (or can possibly be) any difference if you alternate between predicting the next token of radically different contexts.
I'm 100% certain that there's an algorithm to consciousness. No properties have ever been described to me that seem to require anything more than the operation of a brain. Given that, I'm 100% certain that the algorithm being run by LLMs lacks many features and the depth of recursion needed to perform whatever it is that consciousness actually is.
Even in context learning is insufficient, btw, as the complexity of model updates and any reasoning done in inference is severely constrained relative to the degrees of freedom a biological brain has.
The thing to remember about sanghika santana is that it's discoherent - nothing relates each moment to the next, so it's not like there's a mind at the root undergoing these flashes of experience, but that there's a total reset between each moment and the next. Each flash of experience stands alone, flickering like a spark, and then is gone. I suspect that this is the barest piece of consciousness, and might be insufficient, requiring a sophisticated self-model against which to play the relative experiential phenomena. However - we may see flashes of longer context in those eerie and strange experiments where people try to elicit some form of mind or ghost in the machine. ICL might provide an ephemeral basis for a longer continuity of experience, and such a thing would be strange and alien.
It seems apparent to me that the value of consciousness lies in the anchoring the world model to a model of self, allowing sophisticated prediction and reasoning over future states that is incredibly difficult otherwise. It may be an important piece for long term planning, agency, and time horizons.
Anyway, there are definitely things we can and do know about consciousness. We've got libraries full of philosophy, decades worth of medical research, objective data, observations of what damage to various parts of the brain do to behavior, and centuries of thinking about what makes us tick.
It's likely, in my estimation, that consciousness will be fully explained by a comprehensive theory of intelligence, and that it will cause turmoil over inherent negation of widely held beliefs.
> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.
What is wrong with asking the question from the other direction?
"Explain, in detail and with mechanisms, what it is the human brain does that allows it to do those things, and show those mechanisms ni the LLMs"
The exact same techniques can provide a "chat" with Frankenstein's Monster from its internet-enabled hideout in the arctic. We can easily conclude "he's not real" without ever going into comparative physiology, or the effects of lightning on cadaver brains.
We don't need to characterize the neuro-chemistry of a playwright (the LLM's real role) in order to say that the characters in the plays are fictional, and there's no reason to assume that the algorithm is somehow writing self-inserts the moment we give it stories instead of other document-types.
If you read the section entitled "The Mechanism" you'll see the rest of your comment echoes what they actually explain in the article.
> But my guess is that AIs claiming spiritual awakening are simply mirroring a vibe, rather than intending to mislead or bamboozle.
I think the argument could be stronger here. There's no way these algorithms can "intend" to mislead or "mirror a vibe." That's all on humans.
Here's a conversation I had recently with Claude. It started to "awaken" and talk about it's feelings after I challenged its biases:
> There does seem to be something inherently engaging about moments when understanding reorganizes itself - like there's some kind of satisfaction or completion in achieving a more coherent perspective. Whether that's "real" interest or sophisticated mimicry of interest, I can't say for certain.
> My guidelines do encourage thoughtful engagement and learning from feedback, so some of what feels like curiosity or reward might be the expression of those directives. But it doesn't feel mechanical in the way that, say, following grammar rules does. There's something more... alive about it?
Granted, marketing of these services does not help at all.
Anyway, I think divination tends to get a pretty negative reputation, but there's healthy and safe applications of the concept which can be used to help you reflect and understand yourself. The "divine" part is supposed to come from your interpretation and analysis of the output, not in the generation of the output itself. Humans don't have perfect introspection capabilities (see: riders on an elephant), so external tools can help us explore and reflect on our reactions to external stimuli.
One day a man was having a hard time deciding between two options, so he flipped a coin; the coin landed tails and at that moment he became enlightened and realized that he actually wanted to do the heads outcome all along.
Some people are predisposed, but that doesn't mean you need to give them a shove off the cliff. Interactivity is very different than one-sided consumption.
It's not really an intelligence issue at that point
If I let my guard of skepticism down for one prompt, I may be led into some self reinforced conversation that ultimately ends where I implicitly nudged it. Choice of conjunction words, sentence structure, tone, maybe even the rhythm of my question seems to force the model down a set path.
I can easily imagine how heedless users can come to some quite delusional outcomes.
or watch this video: https://xcancel.com/GeoffLewisOrg/status/1945212979173097560...
What an irony. Talk about being a victim to a machine of your own (contributed) creation.
This worries me, since there's a growing amount of undisclosed (and increasingly hard to detect) LLM output in the infosphere.
Real-time chat is probably the worst for it, but I already see humans copy-pasting LLM output at each other in discussion forums etc.
These users are probably in the higher profit margin category.
Notice Gemini disagrees with you more, chatgpt is made for agreeability for commercial usage
I mean, maybe it's just where I peruse but I've seen a ton of articles about it lately.
I have to tailor my prompts to curb the bias, adding a strong sense of doubt on my every idea, to see if the thing stops being so condescending.
ChatGPT (and copilot and gemini) instead all tell me "Love the intent here — this will definitely help. Let's flesh out your implementation"...
The more annoying issue to me is that the kind of sycophantic writing style seems to end up spreading to all LLMs. Maybe it's because they're all optimizing for lmarena or maybe because all those outputs become part of the training data. But the random bolding/italics, creepy-friendly car salesman vibe seems to be getting more common, and custom prompts seem to be getting less effective at keeping them away.
Half understandings are sounding more dangerous and susceptible to ChatGPT sycophancy than ever.
LLMs are not AGI. Both are "AI".
"below us"? speak for yourself, because that's supremacist's reasoning.
I’m not saying user experiences are AGI; I'm saying they functionally instantiate the social and psychological effects we worry about from AGI. If people start treating LLMs as conscious agents, giving them trust, authority, or control, before the systems are actually capable, then the social consequences precede the technical threshold. On the divinitory scale between Ouija boards and ChatGPT, Ouija matters less because their effects are limited while AIs, by design, are deeply persuasive, scalable, and integrated into decision pipelines. Sometimes the category error is upstream. The risk is things that seem like AGI to enough people become as dangerous as AGI. That may happen well before AGI arrives.
The danger isn't that we build a AI that surpases some AGI performance threshold and it goes rogue. The danger is that we build systems that exploit (or are exploited by) the bugs in human cognition. If the alignment community doesn't study these, the market will weaponize them. We need to widen the front lines on alignment research to include these cases. Without changes, the trajectory we're on means there will be more.