What is certain is that LLMs can perform as if they are doing what we call thinking, and for most intents and purposes this is more than enough.
It seems much more likely that they are doing some other behaviour that only sometimes resembles thinking, in the same way that when you press the middle autocomplete button on your phone keyboard it only sometimes resembles conversation.
I find this an utterly bizarre claim given how prone humans are to make things up and firmly insist they did not.
I'm obviously not claiming that "hallucination" is an appropriate term ("delusion" or "confabulation" are probably more apt), but there is something here that is clearly not just a bug, but rather a result of thinking being applied properly but to ungrounded premises. To my eyes, reading an AIs "hallucination" is not unlike reading the writings of a human on drugs, or with a mental condition like schizophrenia, or just of an analytic philosopher taking their made up axioms all the way to an alternate universe.
To be clear, I'm not arguing you've made this claim in bad faith at all.
However, going back and examining my own writing, I have more than once found claims that I'm sure I believed at the time of making them, but that I in retrospect realise I had no actual backing for, and which were for that reason effectively pure fabrication.
An enduring memory of my school days was convincing the teacher that she was wrong about a basic fact of geography. I was convinced. I had also totally made up what I told her, and provided elaborate arguments in favour of my position.
To me this is innate human behaviour that I see on a regular basis. People accidentally invent entire "facts" all the time.
I guess you'll find that if you limit the definition of thinking that much most humans are not capable of thinking either.
The flip side of this is that fundamentally, I don't see a reason why machines could not get the same filtering capabilities over time by adjusting their architecture.
Did you just make this up?
Yes! That is how they work.
Jokes aside, we do produce plausibile sounding stuff all the time well beyond the limit of what we actually know or can prove. I think there is a continuum between formulating statements about things we don't know for sure and we can't prove, guessing details here and there to fill gaps in our memory, misremembering things that we thought we knew, and making up entire facts that sound plausible but are completely invented. Yes, llms seem to have trouble introspecting what they actually know; but it sounds more like a missing skill rather than a fundamental difference in the way they reason.
One problem is that we don't have a clear definition of thinking and my hunch is that we will never have a clear cut one as it falls in the same category of phenomena like alive/death states, altered states and weather systems. One hidden assumption that I often see implied in the usages of this word is that the word "thinking" implies some sort of "agency" which is another vague term normally ascribed to motile life forms.
All in all I think this debate ensues from trying to emulate something that we don't fundamentally understand.
Imagine in a world where medicine has not advanced and we lack any knowledge of human biology, we are trying to create artificial life forms by creating some heat resistant balloon and having it take in and push air. Someone would argue that the globe is alive because there is something in that taking in air and pushing it out that is like what humans do.
Au contraire, LLMs have proven that Chinese Rooms that can casually fool humans do exist.
ELIZA could be considered a rudimentary Chinese Room, Markov chains a bit more advanced, but LLMs have proven that given enough resources, LLMs can be surprisingly convincing Chinese rooms.
I agree that our consciousness might be fully explained by a long string of deterministic electrochemical reactions, so we could be not that different; and until we can fully explain consciousness we can't close the possibility that a statistical calculation is conscious to some degree. It just doesn't seem likely IMO right now.
Food for thought: If I use the weights to blindly calculate the output tokens with pencil and paper, are they thinking, or is it a Chinese Room with a HUGE dictionary?
Eliza is not a Chinese room because we know how it works. The whole point of the Chinese Room is that you don't. It is a thought experiment to say 'since we don't know how this is producing output, we should consider that it is just following rules (unless it is human).
> Food for thought: If I use the weights to blindly calculate the output tokens with pencil and paper, are they thinking, or is it a Chinese Room with a HUGE dictionary?
Well, I never conceded that language models are thinking, all I did was say that the Chinese Room is a lazy way of concluding human exceptionalism.
But, I would have to conclude that if you were able to produce output which was coherent and appropriate, and exhibited all signs of what I understand a thinking system to do, then it is a possibility.
Dictionaries usually provide some kind of useless circular definition. Thinking? The act of making thoughts. Thoughts? The result of thinking. I can't believe people used to pay for these things.
In any case it's something to do with taking input data, doing something with it, and generating new data related to it. That's more or less just recursive inference.