One can, of course, map {base, offset} pointers to the natural numbers (though not necessarily Rust integer types? I don't know Rust) by considering the base to be the most significant bits of the number and the offset the least, but arithmetic on these numbers is not the same as pointer arithmetic in the address-space model.
I do not follow the part, in the "what's in a byte" section, about a byte-by-byte memcpy. Is the question "what is the first byte?" any more or less problematical for a {base, offset} pointer than it is for a multi-byte integer, where place value matters?