1. Hardware vendors don't provide Linux drivers, often requiring people in their free time to create them from often nonexistent documentation.
2. Nobody can agree on APIs and interfaces. Whenever someone disagrees on something, they just fork it and make their own rather than compromise for a standard.
3. There are a lot of low level assumptions made 20 - 30 (or even up to the age of Unix) assumptions (ttys, shell script, group privilege model, kernel control and management of displays / network controllers / usb / devices) that hamper greatly the usability of the GNULinux ecosystem as a drop in replacement for Windows on the desktop. Android gets away with it by throwing out most of that plumbing.
The first 2 are an inherent problem to the entire ecosystem being FOSS. The userbase is the only group that can fix the second, by focusing and standardizing on one way to do it in the most popular distros and hoping it trickles down (pulseaudio, for example). Both are momentum problems.
The latter I just think ads a lot of unnecessary complexity, but you can work around it. I type this on an Arch box running mostly KDE stuff, it can be done.