Knowing the laws of physics doesn't tell you how a brain, economy, or society works - because at every level of scale, genuinely new phenomena emerge that require their own science.
[1] https://www.science.org/doi/10.1126/science.177.4047.393
No, it's not a bunch of instructions, it's a colossal array of vectors that are the outcome of many thousands of lifetimes' worth of stimuli and reinforcement - not dissimilar in (very) abstract terms to the neurons in our brains.
Just because consciousness emerged for we humans and other animals through one mechanism doesn't mean consciousness has/will/can emerge from current LLM technology.
For this extraordinary claim, I think the burden is firmly on those who are arguing that it has/will/can.
If I did all of these calculations by hand, would it be conscious...
That's a powerful argument I haven't seen stated quite that way before..
I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.
But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.
It's basically https://en.wikipedia.org/wiki/Chinese_room – a classic in the philosophy of consciousness and AI.
It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.
It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).
It's not simulating rain if it's making you wet by using sprinklers?
But I guess that was a distraction from the main point: If consciousness emerges from biological processes in the brain connected to the world with a body, why would it not emerge from a simulation of these processes connected to the real world with sensors and actuators?
It seems like circular reasoning to me: The simulation is not like the real thing because it lacks the special thing that enables consciousness (that's Searle's biological naturalism). And it lacks what enables consciousness because it's not the real thing (that's the weather analogy).
Speaking through Zarubin, Dneprov writes that "the only way to prove that machines can think is to turn yourself into a machine and examine your thinking process", and he concludes, as Searle does, that "even the most perfect simulation of machine thinking is not the thinking process itself."
That feels similar to what Chiang is saying. The physical state of being a human is part of human consciousness. An LLM would require many more "levels" that haven't been achieved yet. An LLM with a body, sense organs, physical tools, social dynamics, etc. would all be steps on the path to a "conscious" LLM.
Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.
If you want to get evolutionarily technical, humans are made of cells which began as individual organisms and coalesced into higher life forms. So the concept of a "life form" is very much flexible, so is every capability of a life form, including consciousness.
I'd say AI should not by default gain any social status as human-equivalent even if they become (in whatever regard) more intelligent than humans. But that would require us to drop the notion that intelligence ~= respectability/status/ability to have a full subjective experience.
Most of these kind of pseudo-philosophical controversies actually tell more about the issues with humanity than the new tech/stuff...
So is your brain. That's the problem with this argument.
1. that artificial neurons are very restrictive model for actual neurons, let alone brain/organism, as there is more going on in neurons than a thresholded 0-1 activation
2. a brain does not function in isolation, but as regulator of the bodily functions (along with even more things). If you look at the brain in isolation to the body you don't really understand fully what it is doing unless you narrow your view a lot. Eg the brain modulates production of hormones, which in turn affects stuff like heart rate which then comes back to the brain as signal, in a feedback loop. Not to mention actual behaviour and interaction with the world. Toy models of organisms are not organisms.
3. "interaction between atoms" (or rather matter in general, as we have to take into account electrons, photons, gravity and a lot of other things that matter) is too general, too big, artificial neurons are a very useful (for applications) abstraction inspired by biological processes, but not modelling said underlying biology fully. Nobody can imo right now know if "computation" is a good model for "atom interactions" as in whether we can adequately enough model "atom interactions" in a computationally tractable way, and surely we do not do that right now except in very narrow scopes, and we have good reasons to believe that the current computational paradigms/turing machines are inadequate in doing that efficiently enough.
Does it matter that neurons are more complex than 0-1? Does the fact that transformer layers don't use purely thresholded 0-1 activation invalidate what you're saying?
How do you know that artificial neurons are less capable of producing consciousness than biological ones? How can other people independently verify this?
How does the embodied nature of human consciousness preclude consciousness emerging from a computational system? What is the definition of consciousness if it is precluded from occurring in a computational system but present in biological systems?
Why do you think exactly modeling interaction between atoms matters for consciousness? And where is the fidelity threshold? Is it the planck length?
Finally, a dumb question: how do we know humans are actually conscious, and where is the threshold between consciousness and unconsciousness? And do these criteria exclude all other forms, or other animals?
But calling them "unconscious" is a pretty high bar. Mice are conscious. The house sparrow pecking in my yard right now is conscious.
How do you know that toasters or rocks aren't conscious?
LLMs know more than any human being, are simultaneously experts in nearly every field of science and humanities, are able to make novel mathematical discoveries, can write and understand every major written language, and can give you an intelligent answer to almost any question you pose to them.
How is that not human-level intelligence? If a human could do all of that, we would consider them a genius.
By saying that a computer program is not conscious you are also making an extraordinary claim. You would have to hold an agnostic position until there is a test for consciousness.
You are relying on intuitive obviousness and rhetoric to make the opposing side look ridiculous "how could a TOASTER be conscious, preposterous!", you aren't making a actual positive argument for your view.
I assume that we both think rocks are not conscious, but I'm genuinely unsure of how one could prove this.
And digital circuits are not physically similar to biological brains yet. So we shouldn’t conclude they have the similar consciousness property.
You say this based on what? Your brain is executing instructions on wetware. The entire universe is governed by physical laws.
At its base, the argument that computers can't be conscious is dualist. It assumes that there's some parallel realm of spirit that the brain is peculiarly able to tap into, but which computer programs that function in very similar ways to the brain don't tap into.
> the quality of the simulacrum that LLMs produce vs. pre-ML chatbots.
It seems to me that you're denigrating LLMs, or implying that they're only simulating thought, as opposed to actually thinking. But the difference between thinking and an extremely good simulacrum of thought is meaningless. They become the same thing. It's a bit like the Mitchell and Webb skit about faking the moon landings, in which the plotters quickly realize that the only way to convincingly fake landing on the moon is to fly a rocket to the moon and film the "fake" landings on the moon.
I say it based on the fact that I experience consciousness and therefore know it is a phenomenon wholly separate from the outward effect I have on the world via speech or any other physical action.
>It assumes that there's some parallel realm of spirit that the brain is peculiarly able to tap into, but which computer programs that function in very similar ways to the brain don't tap into.
Ruling this out completely implies that modern science has a total and complete understanding of consciousness and the universe in its entirety, which it does not. I don't think it's unreasonable to leave open the possibility that there is an unexplained phenomenon that explains the conscious experience which is still bounded by the laws of physics.
>.. the difference between thinking and an extremely good simulacrum of thought is meaningless. They become the same thing.
Do they? To me it seems like you equate the process with the result, like saying if I gave you a gift, it doesn't matter whether I bought it or made it by hand, in either case I "made" it because the end result is the same. There is a conscious experience that (hopefully) all humans experience and can testify to, the question is whether or not an LLM predicting tokens based on a giant vector map is experiencing the same thing, which I'd say is obviously not happening. My point about chatbots is about exactly this -- 10 years ago even you would have laughed at somebody that said a hand-rolled chatbot was conscious/thinking, but because the output has improved, now suddenly it's all the same, and your brain is a computer, and Claude has feelings, blah blah.
Calling LLMs "chatbots" at this point just sounds like an attempt to dismiss them. These "chatbots" are now capable of answering any question you can think of more intelligently than 99% of humans. If you don't consider that intelligence, then your definition of intelligence makes no sense.
It’s all just particles, but the higher level differences are vast, and only brains are implicated for first person perspectives via science.