For improved security with X11, try firejail+Xpra, or Qubes OS. They will not just help in the graphical front, but also others like processes and storage isolation.
Gee, if only there were a modern display server protocol with a fixed security model!
Theoretically, X11 is dangerous. But what isn't?