I think the only possibility that comes to mind is creating an ebpf module that sandboxes all filesystem calls and trampolines all ld_open calls.
But then you would have to provide massive amounts of patched/"safe" variants of all kinds of shared libraries which is unfeasible.
But I mean in the xorg use case it would be possible to just provide your own library that fakes the expected returns and sends fake data to the sandboxed applications.
I did a similar thing with barrier (though using LD_PRELOAD, see [1]) on my debian system to force a different behavior.
Source: Am kind of experimenting with ebpf a lot for that use case. C ABIs and SO files are a mess though. A real messy mess.