It has a long a storied history. The concept of running an app remotely and use a local display server is probably the thing that sets X11 apart from the rest of the pack. This is still really useful in my humble opinion -- I can run X11 apps (e.g. xv) on a headless EC2 server running Linux in the cloud and have the UI/display on my Mac (using Xquartz).
This story runs in parallel with the history of Unix itself.