X.org considers hardware acceleration, with XRender, for example. Quite too often, buggy and incomplete graphic drivers and other problems makes it even worse than no acceleration at all. I don't really see these problems as X.org specific, is there any evidence that they are? I think it's more likely caused by very limited resources that are available, incomplete or proprietary (ati and nvidia tends to lag months behind new X server releases, would they ever catch up with a completely new system?) drivers, and other problems, and throwing out X means throwing out all effort spent to solve this problems.

But that doesn't mean that it is not a good idea to explore other possibilities and experiment with another systems in parallel. Like Wayland, for example. Of course it is. Just, don't hurry with burying X11, it's the best what you've got, and likely to remain so for a long time.