This explanation pretty much tracks exactly with model performance on reasoning tasks.
Think about how you reason. Even if you write up the results of a problem you're trying to solve, you don't write down every granular mental step you took to get there, or every line of thought you considered and then abandoned.
https://www.arnaldur.be/writing/about/large-language-model-r...
If there's enough of the "essence of reason" represented in human text, then it's possible in theory some sufficiently large LLM could eventually could grok it and fully generalize over rational thought (i.e, AGI).
Personally I'm extremely skeptical. There's quite a lot of machinery in the brain other than language, and conscious rationality also runs in a constant loop, not a single forward pass through the weights.
I can't disprove the possibility. But if I were a betting man, I'd go all in on LLMs improvements adopting a sigmoid curve rather than an exponential one.