Yes, GPT-4 can at least.
What's interesting here is that base64 does not have the exact same byte sequence (such as a single space) between tokens. So I still think for it to understand base64 is a bit surprising.
If that's the case it would be less accurate for encoding random letters than English.