> 64 bytes isn't that much overhead?

On the contrary, 64 bytes is a huge overhead. The standard maximum transfer unit (MTU) is 1500 bytes. So 64 bytes is 4.2% of the packet (at best!). Add the fact that you need a source and destination address, that's 128 bytes, or 8.5% of the MTU. Or, if you will, 8.5% (or more) of internet traffic, just in port numbers.

Maybe I'm misunderstanding the suggestion, but it would be a terrible idea to have 512 bit port numbers in TCP/UDP packets.