Who says this has to come from the desktop/laptop hardware side of things? They make the fastest and more powerful mobile chipsets (including GPU) right now...
You want to support >1080p 3D graphics projected onto the real world with VERY harsh latency limits (human sensory). Modern GPU's that eat more than 100watts and have terraflops of compute power are only starting to offer the compute speeds VR needs.
With Moore's Law ending I don't see mobile ever being able to render VR settings to a sufficient degree.
Hololens, for example, it's far from being a powerful machine and seems to do it just fine. Obviously we are talking about taking the first steps in a field that is still in development. it's not going to be perfect at first.