There's no inductive bias in the model for it to understand that a particular token is composed of some characters. It _does_ work for commmon words, but it's not super reliable. Transformers are powerful enough to pick up the relationship if it were in the training data, but there's like tens of thousands of tokens, right?

I checked just now a simple letter counting task, it's hilarious: https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7...