Ask HN: Why Isn't It Base128?
Base 64 encodes one of 64 values in an 8 bit character. Couldn't we do better? Encoding 256 choices obviously gets problematic with escape characters that you don't want to send over the wire.
Of your 8-bit character set, ASCII technically includes only the low-order 7 bits (0-127); there are numerous extension character sets that define codepoints 128-255 differently. However, ASCII has a number of non-printable characters in codepoints 0-31; these include things like the bell, field separator, row separator, null byte, etc that will do very strange things to your terminal.
The whole point of base-64 is to create a printable representation of binary data, so if it uses non-printable characters, it's failed in its job. Realistically this creates a range of codepoints 32-126 to work with, but it's much easier if it's an integer number of bits, and the largest power of 2 within those numbers is base-64.