In fact, in almost 2 decades of using Linux every day, I can't remember X doing anything similar. It's always just worked.
Well, we have very different memories then. Sure, X worked reliably once configured. But configuring it was a marathon in hell, as per by my memory, and all the litany of forum posts crying out for help all across the internet. Like, I have at one point had to rescue an install by changing back the config file without a screen at all!
I also fundamentally disagree with the idea that X "does too much", which is often cited in favour of Wayland. The fact that X encompasses everything about how I interact with my computer is one of the things I love about it. I might switch WMs and DEs, but my xmodmap, xresources etc remain the same.
It hasn't needed any manual configuration for a normal setup in ages.
There’s nothing Wayland-specific that would introduce this latency, so I wonder if wlroots/sway have any additions lag.
After upgrading to plasma6 from 5, all the desktop animations have started stuttering. Probably your hardware is too new.