Ah, that's fair.

Still, assuming a lack of other sandboxing, running a keylogger is pretty trivial. It's not by default, as you put it, but an un-sandboxed application has enough access to your filesystem to perform tomfoolery and get your credentials. At that point, there's no need to start a terminal, you just load some shellcode in whatever process you control, and say goodbye to your data.

As for sandboxing, well, you're right, although one could sandbox X11 if they wanted to. "X-on-X" is actually a thing, rarely used outside developing X, but I could see it being used for seamless sandboxing with some work. Work that, inevitably, no-one will actually do, so there's that ;)