What are you using for X security context isolation? I've been wanting a good solution for that one for a while now, and the end of my list is still “write my own isolating proxy” since I never found a good one.
(they have essentially written 2500 lines of c that acts as a proxy of sorts)
If I was worried enough (I'm not), I could use multiple logins. Two different X servers under different users would be completely isolated.
By the way, they may have an history, but nowadays I wouldn't bet on the security of the average distro vs Windows, and I say this as a dyed-in-the-wool Debian user. For example, their Desktop environment does have protections against cross-application attacks, unlike X.