While there are hassles of VR with Android that were only solved by making deep changes to Android, the hassles of PC are greater. And this is a phone accessory that is specifically not designed for changing hardware and software: it only works with the Note 4 running a custom optimized version of Android. But they do agree with you that a dedicated unit is what they want to do as well which is why this is not their own consumer unit.
Also they have a direct line with the phone manufacturer for the display technology and with this strategy of tying in with the mobile world, they can potentially get into a loop of rapidly updating hardware that's subsidized by phone contracts and the like.
The limitations of mobile graphics can and will be solved in hardware and software. It may turn out for instance that something focused on high FPS and resolution can be achieved via low complexity shaders and some novel custom display/GPU hardware.
http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-...