If an LLM wants a message to be read only by another LLM? Base64 is occasionally chosen as an obfuscation method of choice. Which is weird for a number of reasons.
It's trivial to generate a full mapping of all base64 4-byte sequences which map to all 3-byte 8-bit sequences (there is only 8^3 of different "tokens", or 2048), and especially to any sequences coming out as ASCII (obviously even fewer). If I was building a training set, I would include the mapping in multiple shapes and formats, because why not?
If it's an emergent "property", have you tried asking an LLM to do a base48 for instance? Or maybe even something crazier like base55 (keeping it a subset of base64 set).
There is some experimentation on using algorithmically generated synthetic data in pre-training, as well as some intentional inclusions of "weird" data - like CSV logs of weather readings. But generally, it's seen as computationally inefficient - compared to "normal" pre-training done on natural data.
In a world where compute is much cheaper and getting new data is much more expensive, I would expect this kind of thing to be pursued more. We're heading for that world. But we aren't there yet.
I haven't experimented with baseN encodings myself, no. But if I were to down the expectations in advance:
1. Base64 is by far the best-known baseN encoding in LLMs.
2. This is driven mainly by how well represented meaningful base64 strings are in the natural "scraped web" datasets. LLMs learn base64 the way they learn languages.
3. Every LLM pre-trained on "scraped web" data will be somewhat capable of reading and writing base64.
4. Base64-encoded text is easier to read for an LLM than encoded non-text binary data.
5. The existence of a strict, learnable "4 characters -> 3 bytes" map is quite beneficial, but not vital.
How did you get this number? The correct number is 64^4 = 256^3 = 16777216.