Such an architecture could be implemented, it could use one token per letter, or one token per word, instead of the typical 0.75 per word we see.

The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios.

So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.