R10G10B10 matches most HDR displays out there, AFAIK even Windows uses that outside of DirectX (where FP16 is used).
But beyond that...
> Fixing this would require rewriting the entire protocol.
...most of the protocol does not have to do with color values. X11 is extensible and an extension can be used that allows alternative functions that use more advanced color values where that'd make sense. For example, assuming you want to use "full" range color values for the drawing functions like XFillPolygon, etc, you'd want to add have the extended range state in graphics contexts, introduce extended commands for changing it (with the existing commands simulating an SDR color for backwards compatibility). That is assuming R10G10B10 is not enough of course (though because for decades many applications assumed 8bit RGB, it is a good idea to do sRGB/SDR simulation for existing APIs and clients regardless of the real underlying mode of the monitor unless a client either opts-in to using extended color or uses the new APIs).
Another thing to keep in mind is that these are really needed if you want to use the draw primitives using extended color / HDR. However most HDR output, at least currently, is either done using some other API (e.g. Vulkan) or via raw pixel data. In which case you need to configure the window output (a window region, to allow for apps with mixed color spaces in a single window - e.g. think Firefox showing a SDR page with an HDR image) to use a specific color space/format and then rely on other APIs for the actual pixel data.
This is something i wanted to look into for a while now, unfortunately other stuff always end up having more priority - and well, my "HDR" monitor is only HDR in name, it barely looks any different when i try to enable HDR mode in KDE Plasma under Wayland for example :-P. I do plan on getting an HDR OLED monitor at some point though and since i do not plan on changing my X11-based environment, i might take a look at it in the future.