Roughly speaking, the extra features needed to bring a high end phone to Occulus/Vive level is 90fps (make it optional to save battery of course) and USB-C video in. The rest of the tracking sensors/LEDs and whatnot can be built into the headset.
Meanwhile you're expected to pay $700 for a phone and $700 for a headset, each of which has about an 70% overlap in the most expensive components.
On top of all that, it sucks that the headsets for full featured VR vary in comfort and design but are hard wired to the displays. Most reviews say Vive is the least comfortable of the batch but the most immersive.
Ideally the head mount would be a separate thing from the display/processor like in the mobile VR space. That would allow you to find the head mount most comfortable for you, and also to upgrade your VR internals while keeping the same head mount for several generations of technology.
So in my ideal world, VR would be a licensed or open standard system like Dolby whatever or THX are for audio. For example, if you bought a phone with the Whizbang VR logo on it, it would work with your Whizbang VR logo head tracking sensors, Whizbang VR logo head mount, Whizbang VR logo controllers, and be compatible with the Whizbang VR software API. But all that stuff could be made by different manufacturers.