(1) Operating systems have an outmoded security model. Most focus on multi-user security, which still has its uses, but they fail to focus on application isolation. Applications should be installable by anyone and isolated completely from other apps unless specifically granted permission by a user/administrator. The entire malware problem can be laid at the feet of this.
(2) The (related) poor state of installability. Mac has this problem the least with drag-and-drop .app installation, though sometimes even that can be confusing (and .dmg packages are weird beasts... why?). Linux has .rpm or .deb, which applies a massive and complex band-aid to the otherwise awful state of installability on that platform. Windows is absolutely horrible... it's like Linux where you have "installers" that have to do package management instead of a formal package system.
The open source world -- and even commercial vendors that want to keep the PC alive -- have to either address this problem or accept the dominion of the locked-down vendor-controlled consumer compute device. This will require abandoning the old fashioned Unix design philosophy (and the similar way Windows works) and thinking seriously about the problems of installability and isolation. It would be worth taking cues from iOS and Android here, though there's also a lot of room for new ideas.
Oh, and I forgot to mention. If we don't address these problems, all app vendors will pay a ~30% per-sale tax to Apple, Google, and Microsoft in exchange for the valuable service of a platform that provides installability and application isolation. And you know what? The market will pay it, because for most users those things are that valuable.
Point, click, install, with no fear of damaging my system. If I don't like it I click and uninstall. Anything else is completely broken.
It's interesting to note that the prevalence of virtualization is also a sign of the failure of operating systems. OSes in a box (whether via complex container overlays like OpenVZ/Virtuozzo or hypervisors) are an ugly hack to fix the fact that the OS security model is broken even for multi-user operation. The fact that everything requires root to install is the deepest issue, along with the lack of permission structures for things like network interfaces. It should be possible to run an OS and sell accounts to the general public, not VMs, and people should be able to run whatever they want from their local account and this should be safe. The fact that this isn't viable is because OSes are broken, thus we have the huge overhead of virtualization as an ugly band-aid to fix it.