See the ELF rpath, which can be set by the linker. This can be modified after using patchelf.