Does this mean we can skip "tokenization" and feed raw bytes into our models now?
But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world has the GPUs and chutzpah to do pre-training at the long context lengths...)
[0]: https://arxiv.org/abs/2305.07185 [1]: https://arxiv.org/abs/2401.13660
- make vocab ~40k tokens + 256 tokens (for each possible byte),
- start training with 100% tokens,
- then after some portion of train budget has elapsed, randomly replace some tokens with corresponding byte-tokens,
- then ramp the fraction of replacements up so you're training on 100% byte tokens for the last xx% of training, without ever exceeding an 8k (or whatever) sequence length
- then apply the trick from TFA to get xk -> 8xk/64xk bytes of context?
but I'd guess the interesting part of a byte-transformer is multimodality, and we'd need more than a few tricks to get from ^^ to there.
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.