Because it was built in a different time. Back in the 80s and early 90s, X11 provided a way for institutions and companies to mix-and-match their hardware and software environments, each with different capabilities. They would have big iron running expensive software and cheap(er) terminals that could display those applications across a campus.
In a university environment, before there was a PC and network in every dorm room, before laptops were a thing, this was awesome. You could walk up to any workstation or X11 terminal on campus, and immediately have access to all of your applications, files, and everything else. And when PCs and networking came to the dorms, you could get an X server for your PC and do the same.
But those days are long past; the networking use case, which was one of the most important aspects of X11, is pretty much no longer relevant. It's really surprising that Wayland still hasn't completely taken over, and they still haven't figured out how to handle certain interprocess communication that would allow screen sharing and other common desktop functionality.