> I personally think that one reason that the Linux kernel has been so successful was the fact that I didn't have a huge vision of where I wanted to force people to go
I agree. Thing is, this just doesn't work when it comes to user interfaces. You end up with a morass of principalities and wanna-be-kings who all have their own different view of what Linux on the desktop should be. This is an area where consistency really matters (to users).
What's more the primary contributors are engineers who are unrepresentative of any kind of mass user base (potential or actual). People who want to be able to customize everything when no one else cares.
Ultimately I think the pseudo-anarchic development process doomed Linux on the desktop but some external factors didn't help, namely OSX. OSX really is "good enough" for anyone wanting a nix desktop/laptop. Sure it has problems but it means the gap between it and Linux is just that much smaller. And with Linux you give up great hardware (to varying degrees) and a snappy and consistent* UI/UX.
The company that has had the most success putting Linux interfaces in people's hands is, of course, Google with Android (disclaimer: I work for Google). And even there the principality problem has created a fragmentation nightmare as different vendors seek to "differentiate" themselves.
Phones and tablets of course don't have the same expectation and technical legacy that desktops do (from X11 on up). Hell, Apple did the same thing with iOS (essentially forking OSX).
Audio is another problem. Peripherally I've had to deal with PulseAudio and it really is a solution looking for a problem. Graphics drivers are another issue. AMD/ATI and nVidia have of course not helped matters here.
I think Canonical had a chance to make a real impact here but they essentially blew it.