But what would we lose if we just got rid of the notion of bytes and just let every bit be addressable?
To start, we'd still be able to fit the entire address space into a 64-bit pointer. The maximum address space would merely be reduced from 16 exabytes to 2 exabytes.
I presume there's some efficiency reason why we can't address bits in the first place. How much does that still apply? I admit, I'd just rather live in a world where I don't have to think about alignment or padding ever again. :P