There are 6 “a”s in the sentence: “How many ‘a’ in this sentence?”
https://chatgpt.com/share/677582a9-45fc-8003-8114-edd2e6efa2...
Whereas the typical “strawberry” variant is now correct.
There are 3 “r”s in the word “strawberry.”
Clearly the lesson wasn’t learned, the model was just trained on people highlighting this failure case.
hence we continually played whack-a-mole with bugs. we would squash one bug, and another one would appear.
same with llms, squash one problem with a data-fix, and another one pops-up.
https://chatgpt.com/share/6775c9a6-8cec-8007-b709-3431e7a2b2...
Basically one feed forward is not Turing complete, but autoregressive (feeding previous output back into itself) are Turing complete.
Regardless, I’d love if you would explain a bit more why the transformer internals make this problem so difficult?
https://arxiv.org/html/2407.15160v2
The Expressive Power of Transformers with Chain of Thought
https://arxiv.org/html/2310.07923v5
Transformer needs to retrieve letters per each token while forced to keep internal representation still aligned in length with the base tokens (each token also has finite embedding, while made out of multiple letters), and then it needs to count the letters within misaligned representation.
Autoregressive mode completely alleviate the problem as it can align its internal representation with the letters and it can just keep explicit sequential count.
BTW - humans also can't count without resorting to sequential process.
For example, the output for "how many x's are there in xaaax" is 3.
https://chatgpt.com/share/677591fe-aa58-800e-9e7a-81870387be...
https://chatgpt.com/share/6775cb37-4198-8007-82cb-e897220827...