Wayland is just a protocol that is probably implemented over and over and each implementation might have issues. It's mostly likely the different implementations having issues.
Not arguing though, you are right it is just issues: Drivers, the Wayland implementations, how some plethora of apps and libraries have been battle tested then optimized, years over years, for X11. Not as much for Wayland display yet.
It is a common problem with these "simple" things. The problem is complex, and if you simplify one part, you are just pushing complexity elsewhere, you are not making it disappear. It is sometimes a good thing, but in the case of Wayland, it didn't go well.
In any case wayland is not bad if you only have pretty basic needs I guess, some basic things look easier to me there from a user perspective, and troubleshooting x11-related issues for a non-technical person is no fun either.
I think it's safe to assume that it is actually simpler, given that we already have multiple Wayland implementations, but still basically just the one X11 server implementation. Can one or more of those implementations shave off one or two milliseconds overs the next 40 years... Probably yes.
footnote: there is something wrong with the commonly used web search engines and I am unable to find that X11 web demo, I think they are prioritizing recent content over good content. you would think with how interesting that web demo was it was it would show up right away. but I got nothing, so no links. Anyway it was a deep dive into the intricacies of X11 window painting and the author had ported a limited subset of an X server to javascript in order to provide a live demonstration of the topic. I will keep looking.
Found it. https://magcius.github.io/xplain/article/x-basics.html