It's definitely an interesting idea and solves a couple problems, but I'd be worried to have the stability of my system depend on the stability of x-windows.
It scares me the same way windows' http.sys does.
For embedded system you'd not need that much stability.
I'd like to think my car (or POS terminal, or mp3 player) would have better reliability than my desktop.
I already can use XWindow on my phone - I have GPE with XWindow installed on my phone (Motorola A1200): http://gpe.handhelds.org/ .