But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world has the GPUs and chutzpah to do pre-training at the long context lengths...)
[0]: https://arxiv.org/abs/2305.07185 [1]: https://arxiv.org/abs/2401.13660
- make vocab ~40k tokens + 256 tokens (for each possible byte),
- start training with 100% tokens,
- then after some portion of train budget has elapsed, randomly replace some tokens with corresponding byte-tokens,
- then ramp the fraction of replacements up so you're training on 100% byte tokens for the last xx% of training, without ever exceeding an 8k (or whatever) sequence length
- then apply the trick from TFA to get xk -> 8xk/64xk bytes of context?
but I'd guess the interesting part of a byte-transformer is multimodality, and we'd need more than a few tricks to get from ^^ to there.