> I don't really understand how its able to base64 decode values it has never seen though

You are surprised by this but not by its ability to translate languages? You can train a base 64 decoder and encoder neural network trivially, so it is very easy to see how it could have one lurking among all of its many parts give it has likely seen many millions of base64 encoding examples on the internet.