X is a protocol just like Wayland. One of those protocols contains a standardized interface to implement window managers the other doesn't. The argument people are trying to make is that a display stack should have such a standardized interface.
X does not implement everything and the kitchen sink. A bare Xserver won't even be usable without a dedicated window manager, the compositor is completely replaceable and runs as a separate process, etc. Just because it is perceived by some to be bloated doesn't mean it is monolithic.