You're right. There was also an experiment in Meta which tokenized bytes directly and it didn't hurt performance much in very small models.