I think you are conflating composition and prediction. LLMs don't compose higher abstractions from the "axioms, symbols and rules", they simply predict the next token, like a really large spinning wheel.
Said differently, what is prediction but composition projected forward through time/ideas?
Definition: That highly specific, short-lived burst of nervous energy that makes you accidentally drop a small object (like a pen, a guitar pick, or a piece of LEGO) immediately after picking it up.
Exactly. I also only write one word at a time. Who knows what is going on in order to come up with that word.
The most likely series of next tokens when a competent mathematician has written half of a correct proof is the correct next half of the proof. I've never seen anyone who claims "LLMs just predict the next token" give any definition of what that means that would include LLMs, but exclude the mathematician.
Did you read the post that you're commenting on?
It seems wholly believable to me that they are narrow intelligences that are great at some kinds of reasoning and worse at other kinds. Obviously they can reason through problems that most adult humans can't solve