The Abstraction Fallacy: Why AI Can Simulate but Not Instantiate Consciousness
deepmind.google
deepmind.google
1. People who haven't really thought about it, and assume they're conscious because they talk like a human.
2. People who haven't really thought about it, and assume they can't be conscious because humans are obviously somehow special. This appears to be the largest group, and is linked to our religiously rooted culture in which human exceptionalism is the default.
Those first two groups comprise the majority of people, and are not worth engaging with.
3. People who have thought about it, and came to the conclusion that they might be conscious, usually for computationalism/functionalism reasons. This is the group that I place myself in.
4. People who have thought about it, and came to the conclusion that they can't be conscious, usually for biological naturalist reasons. This seems to be the predominant group on Hacker News (among those who discuss it).
The interesting bit to do for both cases is look at the 'they talk like a human' and 'are obviously somehow special' parts, separate the ideas of language, intelligence (memory, fluidity, abstract reasoning), _aliveness_ (as a biological process) and finally ideas about metacognition and theory of mind, and see whether their idea of consciousness as a super-bundle of the above (which is how I assume a lot of default ideas about consciousness are) actually sticks, or whether it falls apart when beings can have a subset of those properties but not all.
Also, I nominate myself to be in the 'People who have thought about it and are becoming more doubtful that I myself am conscious, and the question might be moot.' group.
If you're looking for one of the genuine angles on this:
Consciousness is horrendously under-defined, to the point some people go something like "you know, at this point I figure we'd be better off not having this word at all. "
Some days that's me, with a headache.
[It can be done. But it'll be dirty]
Atoms arranged into a brain generate consciousness. There's no reason to think atoms in other arrangements can't. Brains aren't magic, just well optimized.
That is to say, what evidence would you need from a system in order to think that it's conscious?
But, in fairness, we don't have a science of consciousness yet. Anybody that is 100% confident in their proclamations about this topic is too confident.
It is true that the LLM itself is static. However it's context window is self-modifiable, based on its inputs and outputs.
> I think they need some kind of temporal awareness... and some mechanism for self-modification or active learning based on their input.
Why?! (besides, they do, see above)
I bring this example up, and it's clear evidence in humans that neither of these things are required for consciousness, and one that I deal with in my home. People with dementia that have no memory that are no longer able to learn suffer a different issue that not being conscious.
> If an experience flows through them and leaves them completely unchanged, are they actually conscious of the experience?
This line of thinking precludes dementia patients with no retention of memory are not conscious.
I agree having an experience, and being conscious of that experience are two different things, though.
If human consciousness is reproducible, maybe we will long underestimate the depth and diversity it uses to model reality the way it does.
5. People who has a financial interest in making sure that any eventual AGI isn't granted any kind of rights and continue to be exploited as an inanimate "thing", not as a "being", no matter the actual characteristics of this hypothetical AGI entity.
I mean, take a look a this language in the paper [0]
> This realization pulls the field of AI safety out of the welfare trap. It allows us to focus entirely on the concrete risks of anthropomorphism, treating AGI as a powerful, but inherently non-sentient tool.
This reads as someone that started with this conclusion, and then built an argument to support it.
[0] also discussed in this reddit thread https://old.reddit.com/r/singularity/comments/1sotz9t/google...
If you grant that humans are conscious, then surely domestic cats are as well. It is simply irrational to talk about Claude's "consciousness" without actually engaging with this: cats, humans, pigeons, fish, etc etc all share some common features we associate with consciousness (I don't mean sensory awareness, I mean the fuzzy cognitive concept). Claude really does not. In fact Claude doesn't even have much in common with uncontacted hunter-gatherers! Claude imitates the solipsism of formally educated human philosophers.
It is uncharitable and curmudgeonly but totally scientific to dismiss people in camp #3 as unserious and not worth engaging with: they ignore scientific criticism and don't provide any themselves, it's just a mishmash of sci-fi-adjacent philosophy. There's nothing "functional" about ignoring animals and there's nothing scientific about waving your hands and saying "computationalism." That's certainly how I feel. I know this isn't a very nice comment. But I am so sick of AI folks thinking they can ignore animals and still have an honest conversation about machine consciousness. It's just sci-fi ghost stories.
Are you sure you're a <biological naturalist>? [1] Which is to say, do you adhere to Searle's position about syntax not leading to semantics?
Or is it more like: You're scientifically inclined, and thus you accept Ethology[2] or Neuroscience[3] as being empirically rigorous studies of animal behavior and cognition respectively?
Incidentally, Alan Turing's 1950 imitation game paper was actually pretty Ethological if you look it up. He immediately replaces the question "can machines think" with a more practical operationalization: the famous imitation game.
[1] https://en.wikipedia.org/wiki/Biological_naturalism
[2] https://en.wikipedia.org/wiki/Ethology
[3] https://en.wikipedia.org/wiki/Neuroscience
[4] https://en.wikipedia.org/wiki/Computing_Machinery_and_Intell...
Your response is too condescending to engage with. You should have assumed I know what neuroscience is. Please don't ever email me about anything.
I personally have not been ignoring animal consciousness in how I think about the possibility of AI consciousness and I don't see how animals having consciousness means that AI can't.
My statement is an opinionated position on how we should direct our research efforts and ascertain what is plausible: the behavioral similarities between humans and cats are much more relevant to the question of consciousness than the behavior similarities between humans and Claude, because cats are obviously conscious and that's not true for Claude. The fact that there are almost no behavioral similarities between cats and Claude suggests to me that "Claude might be conscious" is just a ridiculous statement not worth engaging with, even at the level of pre-science. At the very least, the burden is on Amanda Askell and Dario Amodei to explain why nonhuman animals are irrelevant to the question of Claude's consciousness. They have not offered anything like that; instead they seem fully ELIZAed by the chatbot, high on their own supply.
I'm not sure that I agree that's true, and I think that's the crux of the debate here: how do you define consciousness such that it's obvious that a cat is conscious, and why would that definition not include Claude being conscious?
b. There are almost no behavioral similarities between cats and Claude
.
d. Therefore claude can not be conscious.
You are missing: c. Everything conscious must behave like a cat.
This logic is clearly not sound. I don't think you're position is a coherent one.
In the computational functionalist argument, the thing that we share with cats, pigeons, and robots (and in some ways Claude) is the fact that we react to our environment in a way that requires computation.
I myself lean (without confidence) towards weak panpsychism, where a lot of things down from humans to cats to fish to trees to bacteria are in some way sentient. We all have in common a computationally driven sense/"think"/act cycle, and that is where it derives from.
The problem with the "computational functionalist" argument is that a) there's ZERO evidence other animals brains are computational, that is begging the question; and b) pretty much any embedded system is a device that reacts to its environment in a way that requires computation, and none of them have anything close to the psuedoconsciousness of a bacteria. let alone an insect. Point a) is the more important one: only humans have meaningfully Turing-complete brains. Other animals might be hardware-capable but they'll never be trained to correctly execute a program, nor does their own intelligence seem especially amenable to being described by a classical symbolic algorithm - e.g. animals are very good at object identification, quantity discrimination, causal reasoning, and we don't have anything close to a symbolic algorithm for any of these[1]. Computation is linked to the ability to communicate symbolically, and most animals do not regardless of intelligence. The idea that "the brain is a computer" has always been a poetic description, not a scientific fact. It is more correct to say humans have the ability to think computationally because we think symbolically. Again, maybe someone can identify that animals do think symbolically even if they don't communicate that way, or (somehow) we will have a non-symbolic theory of computation. Perhaps a beautiful symphony. Absent either of these two things, "the chimpanzee's brain is like a computer" is simply not scientific.
The supposed "sense/think/act cycle" is just you begging the question again, applying a computational aesthetic in place of understanding; this time it's blatantly false. Animals do not have a "cycle": sensing is an act and processing senses is a thought. Thinking is an act and many animals can perceive themselves thinking (demonstrated in crows and chimps). Dogs think very deeply while they smell, and the manner in which they sniff (tentative whiff versus greedy huffs) is itself an act requiring thought. Most importantly: even in animals, thoughts can be totally disconnected from actions and senses. Actually this might be the most major difference between a pigeon and Claude: their thoughts and actions are not directly tied to environmental stimulus, whereas Claude can only think and act according to a short-term context provided by a human. You can fake an agentic loop with a prompt, but it's not convincing agency the way a nematode has convincing agency. It's just a chatbot in a loop. If you expose it to real sensory data like a webcam, the agentic behavior becomes even more brittle and unconvincing. It's just nothing like an animal.
[1] I know there's work being done on formal causal reasoning, I thought this monograph was interesting: https://direct.mit.edu/books/oa-monograph/3451/Actual-Causal.... I am not convinced by it. The funny thing about these causal theories... they don't have a causal explanation :) :) :) The argument works by going through cases until you agree it works, empirically, possibly after complicating things further by patching out oversights and inadequacies. Very amusing. Causality is a tough nut to crack!
Obviously, language-driven thought is not a requirement for consciousness, not just in other animals, but even in humans. The thinking mind takes a secondary role in ordinary daily human life. The truth is that a human being behaves the way they do is not because of thoughts, but because of conditioning (the thoughts are not the primary driver of decisions, actions and behavior). The 99% of the action and responses are trained, the thoughts that we have are also part of this conditioning (most thoughts are unconscious and they are inter-wired with the behaviors, even a seemingly conscious self-reflection thought can be an automated pavlovian trigger). For example, one may think that they get up and go to work because they have a thought "I have to get up, now I am going to go to work", this is an illusion and complete misunderstanding of what consciousness is. Or one can have a psychological insight about oneself, if it's repeated and follows a behavior consistently, the very thought is just the equivalent of whistle-salivation. The thinking mind gives us that 1% to self-reflect, adjust our behavior, learn, predict the future and that differentiates us from other mammals, it's a powerful tool, but just a tool, but it should not be confused with consciousness and it should not be confused with the mind as a whole (in the materialistic sense). The way our brain functions is anything but like an AI agent. And what is consciousness? It's not the thinking mind. It's the experience. It's the direct perception of the senses. The consciousness is what is seen, heard, smelled, touched, thought (the experience of having a thought) in the moment. When you practice meditation, you get to discover the consciousness directly by becoming separated from the thinking thread. The thinking thread becomes more like an external tool, like a computer inside you and you realize directly that it's just a part of the cognitive faculty that makes you navigate your life, not the entire thing.
The LLM (and the harnesses) as built right now merely simulate the tool (the thinking mind). It's not that because this is some code ran on a beefy, but regular piece of tech invented in 20th century you may have at your desk that it does not have awareness (that's also a good argument), but because the way they function and operate is nothing like human (or mammalian) brain, then why would you think that regular code running on a regular PC could gain awareness? My point is that there's no similarity argument, LLMs, despite all their incredible capabilities (to threaten our jobs), are not remotely similar to the way our brain works.
Secondly, even if someone built an artificial brain made of whatever that simulates the biological structure, because of the philosophical zombie problem (the fact that there's no way to scientifically observe consciousness), you could never be too sure if a key ingredient was not missing and you are looking at an NPC. The consciousness is not a property of the physical brain, it's literally immaterial, it's the direct experience of the senses. You can make an optimistic assumption that every person and animal experiences consciousness the same way you do, but there's no way to rationally accept this assumption for anything created artificially.
It really has 1000 meanings. Usually whatever the speaker wants it to mean.
My very amateur view is that until the underlying compute architecture and substrate resembles artificial biology more than silicon, we wont get there.
The latest advances in AI have given me even more appreciation of biology and evolution. It's incredible what the human brain can do with about 20 watts of power, barely enough to power a lightbulb, in comparison to what it takes to run even our most basic LLM models.
I say clearly as at some point we reach proof by construction. As in, we already built intelligence because the system already completes tasks that require intelligence.
We are so far into what would have been science fiction five years ago and the goal posts have moved so far.
For anyone who disagrees, I challenge you to prove deep learning systems cannot solve <task with specific outcome humans can solve but not AI> given sufficient data and compute.
I think the strongest sign we have true intelligence already is no one has built any benchmark that AI cannot solve.
Yes, our current robotics lags AI, so we don’t have the equivalent of the human body to give our deep learning systems. Thus, it’s expected AI will be limited in physical scenarios.
Second, hallucinations are present in humans. We are highly biased to ignore all the misspoken words in everyday life as we have error correction built into normal conversations. How often do you have to have someone repeat or rephrase something?
It just doesn’t make sense to me.
It’s like there are people out there whose belief systems are incompatible with this tech existing.
Sure, it has limitations due to training data. It has limitations with no physical body. It cannot combine training and inference the same way a human does. But none of those are measures of intelligence or required to be intelligent.
> Consciousness is a property of humans biology
You're assuming consciousness is a product of biology rather than attracted to biology.
Sure, but people have built benchmarks that no AI constructed before the benchmark was released can solve. If I know the answer to a benchmark problem, I can construct an "AI" that can solve it on a note card.
But I don't think the takeaway is "humans are intelligent and LLMs are not", it's that our vocabulary for talking about the intersection of language, cognition and compute is not up for the task.
No true Scottsman fallacy.
> Consciousness is a property of humans biology
Just because we only observed something in human biology, doesn't mean that it can't be found elsewhere.
I mean being water based is also a property of human biology. They share this property with other things like lube and chicken soup.
It is not just uninteresting that computer programs can be written to accomplish information tasks, it's intellectually dishonest to anthropomorphize machines and algorithms to characterize it as consciousness.
> no one has built any benchmark that AI cannot solve
"Be human."
My point still stands
The crux of my argument is Consciousness is irrelevant to any AI debates. It’s not necessary to perform tasks we previously deemed only humans could do.
This approach actually makes testable (and tested) scientific predictions.
This makes Searle-derived papers super-weird for me; since from my perspective they seem to disprove the existence of life. (and it makes the name of the philosophy "biological naturalism" very ironic to me :-P )
(for extra irony, Turing actually went into biology late in his life. See: Turing 1952 "The Chemical Basis of Morphogenesis" )
Are we just autocomplete machines with sufficient enough variable pseudo-randomized input?
Total drivel. Consciousness in biological systems is "a given" because of metabolism?
That doesn't mean I agree with Searle though!
It depends on how the question is asked. Again, the wording is very terse so I can't determine what the people thought they were answering. Possibly you have a better insight?
if you were to try to stage it as an actual scientific experiment it fails to hold up. No control (there's just the one room), no single or double blind (the researcher self-reports) , and badly defined elements (the contents of the notebook are not specified). Of course you reach the conclusion that room doesn't understand.
Compare to Turings imitation game experiment: Two participants (principal and control if you will), Double blind (they're in closed rooms so the judges can't see them), and you can have multiple people doing the scoring. We can conduct this IRL, and in fact if you've used IRC or discord, it's almost a natural experiment there.
Looking at it as a scientific experiment isn't so strange, Searle is responding to an executable experiment with an intuition pump. Why should the latter win?
Note that there are many ways to cut this, but this is one of mine.