I always thought part of the point of open source is that the code is auditable so you can see if anyone does anything nasty. That obviously doesn't mean you can grok the code of all that you use but at least the community may in theory be able to. If you're installing software you don't trust, then in my opinion you've already lost most of the battle.

That said, I would think mandatory access control via SELinux could do a lot of what you're suggesting, though I've only ever really tangled with it on servers, not consumer/desktop type computing. I think in that sense, Linux actually got a lot of security features before Apple/Microsoft.