For example the TI Explorer Lisp Machine came with an X11 server written in Lisp. On my Symbolics Lisp Machine I used the usual MIT X11 server written in C - this was possible because the Symbolics Lisp machine had a C compiler.
But so as I recall was the virtual machine that interpreted the bytecodes the compiler produced.
Here I'm referring to what I remember being told about the CADR's microcode (it had a LOT, with options for more if you were willing to buy more $$$ static RAM), but the others were all direct descendants and I gather did pretty much the same things.
Would be interesting to see how much is needed to get it running with some other gl context. Also -- maybe this'll run under OS X and windows without requiring a X server for the GL context?