If Apple has shown something, is what putting millions of dollars and more than a thousand full-time engineers working on a Unix OS with custom hardware looks like. Most Linux users don't care about the X server nor find it underperformant. Hell, not even GUI developers have to touch it.

I would argue that Linux isn't mainstream because it has nothing like Apple's budget, specially on design and marketing. Add to that those hardware vendors that are actively hostile to support from other operating systems, and software developers that don't care, and you have it. Still, Linux is doing fine, for what resources it has.