> but for some reason it doesn't seem like the right approach.

The approach being used in containerization is namespaces. You can put new processes into a new IPC / user / PID / network / time / etc. namespace, which isolates them from the parent namespace. Once that's done and you can't mess with other processes via the filesystem / kernel, the remaining hole is servers with inadequate security models, such as X11.