No, the hard part is building a good API and user interface that works for everybody. IMO that's mostly why there are a lot of half-finished and inconsistent things like that X11.
Option 2: replace X with something entirely different -- different rendering, different input, different IPC, different organizing principles, different programming models -- and patch all downstream dependencies to use it.
If you can't see that Option 2 is clearly more work and more disruptive, I don't know what else to say to you.