Because that would require OS level integration with the phone.... Even with full screen apps, swiping in from at least one direction is usually the "escape" ... Also, phones come in many form factors, and creating face adapters for multiple devices would be difficult, to say the least... though options for 3d printed services may alleviate some of that.

What I'd really like to see is a Bluetooth standard for full remote input and ui rendering. That would be useful for car stereos as well as with devices such as this.