I'm biased, because I did work with IRIX at SGI and I did in fact touch some kernel code as well.
What I miss from IRIX, that no other system has yet replicated: 1) Realtime mode. RTLinux doesn't count. In IRIX, you could run your own code at a higher priority than the scheduler itself (this was called hard realtime), and you'd give it time slices when you could, or a core or two. 2) Frame scheduling. When rendering 3D, you could have the scheduler connected to the monitor refresh rate to make you less likely to miss a frame boundary.
Yes, Motif was very plain, and X11 was a hot mess to work with, but the thing had capabilities which are still hard to find.