I would like to see what happens when you go the other way. Extremely naive tokening, for instance none at all. Just a stream of bytes or nybbles.

It might take far more training but also it might avoid any biases introduced by tokenisation.

[edit - see @api had the same question]