On Linux/X11 even when you run a program sandboxed or as a different user if you use a master Xserver the sandboxed program still can listen and modify all your input/output including keyboard/mouse events and window content of every application.
There were various attempts to improve this situation in the early 2010s, typically using Xnest or Xephyr in conjunction with other sandboxing techniques. I believe Qubes OS followed that approach but it was awkward, limited, had major performance problems, and yet never managed to fully prevent circumvention.
The fact is that the X model was never designed with these threats in mind.
Not sure what you are talking about. Qubes offers reliable protection with decent performance, unless you work with graphics. See also: https://news.ycombinator.com/item?id=49678250
But then you would have to provide massive amounts of patched/"safe" variants of all kinds of shared libraries which is unfeasible.
But I mean in the xorg use case it would be possible to just provide your own library that fakes the expected returns and sends fake data to the sandboxed applications.
I did a similar thing with barrier (though using LD_PRELOAD, see [1]) on my debian system to force a different behavior.
Source: Am kind of experimenting with ebpf a lot for that use case. C ABIs and SO files are a mess though. A real messy mess.
There are ways to force sandbox jail. For instance, giving processes only a partial view of the computer system. GoboLinux did this years ago via ViewFS (https://linuxphilia.blogspot.com/2009/07/gobolinux-is-linux-... search for ViewFS). There are many other similar solutions, some probably better.