How secure is this (when also running the GUI using Xephr)? Compared to say running Virtual Box VMs or Quebes or LXD virtual machines?
EDITED in response to a comment.
EDITED in response to a comment.
It is as secure as the individual components, that is whether there are security vulnerabilities in LXD, and in Xephyr. There are currently no pending security vulnerabilities to fix in either (as far as I know).
Of course, the same goes with VirtualBox. It is as secure as there are no pending known security vulnerabilities (https://www.techrepublic.com/article/10-new-vm-escape-vulner...).
Is this sort of thing possible with Wayland? If so does Wayland already enforce the necessary process isolation or does something like Xephr for Wayland need to be developed first?
Xephyr is the appropriate tool for what we do (it's a display server for the X11 protocol, can reuse the acceleration of the desktop). There are equivalent tools for Wayland, it's just not required yet. It is possible though to do these things with Wayland as well and probably there is already a tool that I am not aware of.
There is already process isolation with the containers. The lingering issue is with the graphical output on either X11 or Wayland. That needs some extra care. With X11, choice is Xephyr. With Wayland, there should be something equivalent and is probably simpler.
In terms of security of Xephyr, there is an issue. It is a tool that is not used very much and may have some unreported security vulnerabilities. But the same goes with qemu, the hardware emulator. qemu is big and has too much functionality which makes it likely to have yet unreported security vulnerabilities. Have a look at https://cloudplatform.googleblog.com/2017/01/7-ways-we-harde... which specifically mentions the risk in points 2 and 3.
Nevertheless, it should be very important to also implement an option of using Xephyr as part of the application isolation efforts.
If you want to test programs that might be malicious, then you would set up a separate X11 server like Xephyr and get the output to get directed over there. In that way, both the filesystem and X11 session would be separate from those of your host.
I mention the use of Xephyr in the Conclusions of the post. There is some new functionality in LXD that is being released soon that will make it very easy to use Xephyr as well.