The fact that LLMs are usually trained on tokens and not on characters, doesn't really speak about what generative AI is going to reach or not.
>most people selling LLM today pretend it is AGI
Who are these "most people"?
>most people selling LLM today pretend it is AGI
Who are these "most people"?
The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios.
So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.