Limited use of registers in x86 does sound like a good explanation for it being a wash in terms of speed
It seems like there is very little language-level support for small pointers. Only the unreleased Jai language seems to have "relative pointers" (or maybe they took this out)
The Zig compiler recently replaced pointers with integers in their code representation for memory/speed, but lost some type safety in the process -- i.e. a T* isn't a U*, but indices into t[] and u[] are the same type. I'm not sure I like that tradeoff, and it seems like language-level support would help