Also, it seems that current LLMs suffer from the quirks of tokenization, hence they can't spell letter by letter or don't know anything about pronunciation. I wonder if efficiently training character level LLMs is the way to go.
Since all the letter counting/spelling/rhyming issues are pretty much irrelevant for the vast majority of practical text understanding tasks as they pretty much trigger on toy puzzle problems, it makes all sense to spend any available computing power on increasing model size (which helps all tasks) instead of switching to character models; when some niche use case does need character analysis, that can get a specialized model, but we don't want to pay so large overhead cost for every task.
But I believe there is a lot of room in exploring better tokenization. An obvious example some models are playing with is how to tokenize numbers. In GPT3 10000 is a single token, while 10001 is two tokens (100-01), which obviously makes it harder for the model to learn math. Some models improved on this by special-casing numbers in the tokenizer to turn each digit into one token. I think GPT4 settled one one token for ever three digits.
For pronunciation, maybe throwing some IPA dictionaries into the training set might already do the trick. The model can already infer facts across languages, so if we can somehow teach it a phonetic model it might be able to use that.
I checked just now a simple letter counting task, it's hilarious: https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7...