>Algorithmically it seems to be a 8-bit byte to character mapping

Base64 is 6-bits, so there is no mapping of a single character to another single character. Changing 1 character in the input can change 1 or 2 characters in the output because each of the 6-bit "characters" are up against each other. So there is no mapping of input character to output character(s), it depends on context.

I'm aware that regular NL translations are a similar problem, but with base64, it's a much more exact and unforgiving problem, and GPT-4 seems to perform at a very high level.