X was developed on VAX, 68k, and later SPARC and then x86. VAX and x86 are little-endian, while 68k and SPARC are big-endian. Portability across these platforms was important and this was the inspiration for the RPC (now gRPC) platform-independent external data representation (XDR) standards.

Things were still not perfect with portability even back then. I remember taking a class in OSF Motif where everybody in the class was using SPARC while I was using x86 Linux. The code was somewhat portable between platforms, but many of the arguments to the (Metroworks) Motif library functions needed their endianness swapped in order to work properly on Linux, so the code examples from the course would not run on Linux without modification.

I realize that there may be some performance improvements realized by removing cross-platform compatibility, but I do not believe that the performance gains are worth the loss in platform compatibility. If X.org does this, it will eliminate one of the core reasons for sticking with X vs. Wayland.