But then you would have to provide massive amounts of patched/"safe" variants of all kinds of shared libraries which is unfeasible.

But I mean in the xorg use case it would be possible to just provide your own library that fakes the expected returns and sends fake data to the sandboxed applications.

I did a similar thing with barrier (though using LD_PRELOAD, see [1]) on my debian system to force a different behavior.

Source: Am kind of experimenting with ebpf a lot for that use case. C ABIs and SO files are a mess though. A real messy mess.

[1] https://github.com/cookiengineer/barrier-disable-dpms

There were various attempts to improve this situation in the early 2010s, typically using Xnest or Xephyr in conjunction with other sandboxing techniques. I believe Qubes OS followed that approach but it was awkward, limited, had major performance problems, and yet never managed to fully prevent circumvention.

The fact is that the X model was never designed with these threats in mind.