How? Maybe this should be better documented & recommended. I suppose at some point you're just statically linking with more steps - though for a problem like this it might be worth it.
See the ELF rpath, which can be set by the linker. This can be modified after using patchelf.