I don't get how that changes anything.

You still need to go through the memory mappings every time, which is also randomised thanks to ASLR.