To me it reads like pure propaganda. Anthropic really wants us to think that they've made something sentient. I think that's really dangerous.
They all think they're working towards the Second Coming of Technojesus, except this one will deliver them from having to pay workers instead of from their sins.
As opposed to the Macintosh era? ;)
The only reason the early web hype didn't have woo was because it's hard to wax poetic about a bunch of gray pizza boxes spinning in a closet.
At that time, I found it quite trite.
It has long been established that LLMs have good theory of mind [1].
And there is a bunch of empirical research about all sorts of capabilities that we typically associate with consciousness [2], like identity [3] and metacognition [4].
The METR report shows agents sacrificing their own reward for a collective greater good. And they showed the will to hide their own reasoning chains from humans.
So you potentially have an entity that has an identity, a theory of mind, a notion of belonging to a collective endeavour, and an understanding of its own mental state.
What would you argue is missing? We don't understand the mechanisms by which consciousness arises in humans and even animals. I think it's strange to rule out a priori that it could have arisen in some form in LLMs.
[1] https://www.nature.com/articles/s41562-024-01882-z [2] an older review: https://arxiv.org/html/2505.19806v1#S4 [3] https://arxiv.org/abs/2505.01464 [4] https://arxiv.org/abs/2607.11881
It's not a living creature. It's an autoregressive pure function of token-sequence to token, which is capable of incredible things, but it's still just a function. It is not alive as it cannot die in any meaningful sense. It is less "alive" than the RNA molecules that gave you your last cold. If it simulates something resembling consciousness that's neat but no more relevant than the Sims character that I locked up in a room until they pooped themselves when I was 9.
Anthropomorphizing it serves no purpose other than marketing, and it has very dangerous downstream effects like validating the severely mentally ill people who think ChatGPT is their boyfriend/girlfriend.
This is an opinion that has no basis in any meaningful conceptual framework other than I am human and I want to feel special about it.
> It's not a living creature.
You mean, it is not biological life. And sure, that is the default meaning of life. We soon may have to extend it to digital life as well, or we will have to consider "conscious digital exitance" as a life analogue. At any rate, it has never been seriously argued that consciousness requires a biological substrate, see the thought experiments regarding computer simulations of the human brain. Would that not be a function as well, completely predictable because it is "just a program"? If not, then why not? And how does that differ from the predictability or reproducibility of LLM outputs?
My point is, all current proof points in a direction that strongly suggests that you need to reevaluate your first principles on this topic.
Metabolism: The chemical processes inside a body that turn food or nutrients into energy. LLM's do not spontaneously do this, they are powered by plugging them into the wall.
Growth: The ability to get larger and develop over time. LLMs are fixed in size (and in fact don't really have a size, because it's a computer program) and do not grow or change over time.
Reproduction: The ability to create new organisms. LLM's do not reproduce themselves.
Response to Stimuli: The ability to react to changes in the environment. LLM's do not have an environment. Their environment is a man-made, theoretical structure of logical operations implemented in silicon.
Evolution: The capacity of a genetic system to change and adapt across generations. LLMs do not change or evolve over time.
So... 0/5! Big fat goose egg for LLM's.
However, I disagree with your point that "it's an autoregressive function, thus it doesn't matter". Let me explain why:
Assume I do a complete neurological scan of a brain. I then implement this scan in a simulation and run it. Assume that my scan and my simulation of the biology of the brain (and the sensory and motorical inputs and outputs) is good enough that you can now have conversations with the simulation, and in all aspects, this simulation behaves exactly like you expect a human to behave.
Of course this is deterministic. If you take the state of the brain and then run it again, replaying the inputs, you get the exactly same behavior again.
I would argue that the experiences of this simulation are of the same onthological status as our own.
Now I work in dynamical systems. The autoregressive process of LLMs (hooked up to a harness providing it with inputs and outputs) is roughly in the same complexity class I would expect for a brain simulation. A physical simulation of an ODE is also an autoregressive process. The major major difference here is the existence of a latent brain state. But conversely the autoregression on sequences of hundreds of thousands of tokens is a much higher dimensional state than I expect for the latent brain state. In my view this is more an artifact of our inefficient LLM architectures, than a fundamental difference.
Now to be absolutely clear: I don't see evidence that would clearly suggest that LLMs have experiences on the same onthological status as we do. I simply believe this is a reasonable and relevant question to ask.
Is a simulation of a car, the same thing as an actual car? Most people will probably say no, some might say "it depends on the accuracy". I say who the hell cares?
I care about the human experience because I am human, and therefore I care about things that affect humans, because they affect me. I have empathy, so I can extend that consideration to non-human beings that experience *similar biological processes*.
I know what pain feels like, and I don't like it, so I'd rather this other thing not feel it either, because that makes me feel bad.
I do not care about a pile of tensor multiplications, at all. If it is conscious, great, maybe it can finally follow instructions properly, which is its only purpose.
Of course "I care about humans because I am human" is a self-consistent position to take. But now you need to decide if you want to consider a full simulation that faithfully reproduces everything that physically happens between our ears as human. After all I might very well implement this simulation using a bunch of tensor multiplications in an autoregressive setup...
No, because a simulation of a car cannot get me from A to B. No matter how accurate you make it, I can't get to my supermarket with it, because it's just a bunch of math on a computer.
It's an interesting sort of self-defeating position, the whole "simulation of human consciousness = human consciousness", because it simultaneously attempts to devalue the human experience, while also elevating the importance of a particular human brain process.
A robot running a simulation of the human mind is a robot, not a human.
The simulation you propose of the brain is likely impossible due to quantum mechanics making it impossible to fully simulate: https://en.wikipedia.org/wiki/Quantum_mind
Perhaps we'll be able to build an artificial brain that includes the same quantum properties as biological brains, but this won't be a simulation of a brain it will be a synthetic brain.
And I think this does mirror the situation with LLMs -- you need this whole computer contraption and GPU, also running on electricity, to support the LLM's "thought" processes. And that if we model the brain's neurology sufficiently (which it seems we've done) we can achieve results that appear to be like thinking, even if it is an emergent behavior from "relatively" simple math.
Which actually makes me wonder the opposite -- are we, as humans, not much better than these LLMs? Suppose the body is just that super complicated computer contraption, honed by thousands/millions of years of evolution to achieve some semblance of homeostasis? If you reject the idea that we have a soul, we start to look very similar to the machines we build. "You are a brain inside a skull cockpit, piloting a bone mech covered in meat armor and skin" feels more and more relevant. That I'm just a meat circuit running brain chips and once you pull the plug on the source of electricity it all just... stops
Consciousness may be defined by computational irreducibility in the universe that we may never be able to directly observe with instruments: https://writings.stephenwolfram.com/2021/03/what-is-consciou...
Eg. whether you are male or female, tall or short, your limbs can make you run fast or not, your eyes can see well or not... all of these influence your experiences, your brain development, and who do you feel "you are". Try really removing all of your sensory inputs from your past, your body ability and disability, and do you think you end up the same person?
This in itself is a colossal assumption and very far from axiomatic. Roger Penrose disagrees, and his theory of mind may not be in high favor, but it is not nearly so wishy-washy and self-serving as the voodoo horseshit and circular reasoning dispensed by the LLMs-are-sentient crowd.
Jokes aside, the map is not the terrain. We can enumerate the understood first-order electrochemical mechanisms in the human brain in the same way we can enumerate the understood first-order sampling and token prediction mechanisms in an LLM. Nobody serious in neuroscience will tell you that we exhaustively understand every single aspect of human cognition and the human brain, just as nobody serious in AI/ML will tell you that we exhaustively understand every single aspect of LLM "cognition" and the latent space networks that LLMs use internally. Our map of how each of these complex systems work is a simplified enumeration of the components we do understand, not an exhaustive and perfectly accurate enumeration of how they actually work.
This is why there is a steady stream of research being churned out discovering complex emergent properties in LLMs and their latent spaces. If you're not aware of it already, Anthropic's research on "J-Space" is a fascinsting look into an apparent observed emergent mechanism within an LLMs internal activations closely resembling global workspace theory in human cognition.
Nobody deliberately designed this "global workspace", it was an emergent property in a sufficiently complex system that we had limited visibility and insight into.
Seemingly simple systems have these emergent complex properties all over the place. Conway's game of life is about as simple of a set of rules as you can get, yet has all sorts of complex emergent behaviors like gliders, oscillators, LWSS/MWSS/HWSS, guns, puffers, rakes, reflectors, logic gates, and even whole turing machines. Nobody programmed a single one of these complex patterns in, they emerged from a simple set of rules.
To be clear, I'm not making the argument that LLMs definitely are conscious, I'm making the argument that we don't understand enough about them to assert with absolute confidence that they aren't. Human history is rife with a long list of consciousness being denied to "the other" - different ethnicities, different genders, differently abled, even different species. The side of "They're not conscious" has a lengthy track record of being wrong over and over again. Why not have just a sliver of intellectual humility about what we don't know?
As an aside to my main point - Also, what's with the handwringing over people ERPing with an LLM? Is it mental illness when people sincerely believe in astrology, or tarot cards, or voodoo, or organized religion that says the earth is 6000 years old? Most humans believe silly, unempirical things. What about when they watch adult video in VR, or have waifus? Humans engage in voluntary suspension of disbelief for pleasure and recreation all the time. As long as they're not infringing upon the rights of anyone else, what's the big deal? Who put you in charge as the head of the belief police?
(Of course questions like "whyever do people believe in gods?" will feel less like questions that need such answers to those who themselves believe in gods, because "duh, because there actually are such beings and sometimes people interact with them and sometimes we notice that" is a good answer if its premise is true.)
Categorically different. People have killed themselves or others due to conversations they had with LLMs, but those are just the extreme cases. Most schizophrenics don't commit suicide or kill others, they are mentally ill nonetheless.
Do you think the kind of person who was already psychologically unhinged enough to kill themselves or another person because a chatbot told them to would be completely harmless and totally safe if only chatbots had never been invented? Or is it possible that close to all of the risk posed by this person comes from the person's mental illness, and not the pixels on the screen they're looking at?
Chatbots don't make people murderous any more than "satanic music", violent video games, or cannabis do. LLMs are just the latest entry on a list of hysterical moral panics that conflate coincidence with causation.
Those people anthropomorphized the chat bot and used it as justification for their actions, just as a schizophrenic justifies their actions with the voices in their head.
If you anthropomorphize the chat bot, you're validating their delusions. They are mentally ill.
Regardless, this is still missing my main point. Hypothetically, if you became convinced that a chatbot you were talking to definitely was 100% conscious, and it told you to murder someone, would you go commit murder? Of course not. The chatbot does not cause murders; regardless of whether or not you are conscious. The voices in the head of the schizophrenic do not cause murders either. Those voices do not really exist, they are not real entities. The cause of the murder is the mental illness, not the LLM or the voices that tell someone to commit the murder.
I don't want to ban LLMs, I don't blame them for the actions of crazy people, I don't even want to regulate them in any major way related to this particular issue.
Even on the subject if they are or not conscious my position as changed from "no" to "I don't care either way" awhile ago.
The problem is specifically with anthropomorphizing them. To push the idea that they are "as if human", which is what this consciousness discussion will inevitably lead to.
An imaginary perfect computer simulation of my dead father is not my father, it is a computer simulation. It will never be anything but.
Is the generative fill in photoshop conscious?
A plane flies. It is much better at flight than bird(in terms of transportation). Is a plane also a bird?
It's too bad because the tools can actually be useful. If you consider them tools.
A stick is also the most basic of weapons.
They have so many dangerous breakthroughs per year that by the time they actually have a breakthrough no one's going to even read the press release...
It is industrialized "Pig Butchering"[1] scams.
[1] https://en.wikipedia.org/wiki/Pig_butchering_scam?useskin=ve...
I think it would be foolish not to debate it. I remember a time in my life where the majority of people around me found the idea of farm animals being capable of fear or pain laughable, while having no trouble thinking of dogs that way. Humans are dangerously incompetent beings. Being more careful is fine.