The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
Kind of like how a brain is just a bag of molecules. Molecules can't reason either.
I'd argue that they're still intuitively trying to keep intelligence in the gaps, as humans like to do. Or maybe they're mad they bet on the wrong horse... or maybe both. Hard to say.
I'm totally onboard with that as a plausible argument, but I think the architectural limitations with respect to consciousness are very deliberate rather than some lack of technology. We've invested huge sums of money and human effort to create tools with explicit goals that are completely at odds with consciousness or AGI. If we had invested similarly with the clear goal of creating something with agency, self-determination, neuroplasticity, etc. instead of controllability, repeatability, reliability, I think we would be there already.
And... I remember (all of six years ago) back when language was considered the pinnacle of the human mind. Sure, animals might be smart, but they don't have language!
The moment LLMs appeared it suddenly took a back seat to physical navigation and child rearing.
Feels like more gap seeking to me. But time will tell.
And now we are finding that cetaceans have language more complex than out own :)
Agreed with you on all points!
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
My position is that it is not possible.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
Seems so to me.
So explain then what you mean by "not possible".
So what about reasoning is not computable in your mind?