The X11 part would be solved by xpra. Capsicum, however, feels like the better tool for the job of privilege isolation here. One of the more important things is that you're able to lock down access much more finely than you can with docker. You can use this to ensure that the locked down application can only communicate with xpra and put files only in certain folders, for example, without being able to see or interact with other processes.

Your approach has the problem of needing the inherent insecurities of docker. Because everything within docker has to be managed either by root or someone within the docker group, you have a greater surface area exposed where if an malicious app is able to get hold of the docker socket file, it now owns your system. A capability-based security system, on the other hand, wouldn't be able to touch the docker socket, even if it was run as root.