Most of the graphic drivers is already in user space. What's in the kernel is there the to manage memory and handle coordinate command submission and scheduling as well as display output configuration. There is no reason why this shouldn't be in the kernel.