Also, a well made VR viewer that attaches to a phone (i.e. Gear VR) is just as big and almost as heavy as a stand alone viewer would be. So there are only marginal space savings to be made.
Making the best VR experience is not the same thing as making a mobile phone. Screens needs to be optimized differently, possibly using more than one screen. Chip cooling is a thing in VR. Faster sensors. Different appication stack. Support for positioning and control input. So many things are different from what needs to go into a phone.
Compared with desktop/console, the mobile eco-system is 10X the size and revenue and longterm this is where you want to build your product. Desktop and console VR, were the headset is attached with a cable to a big loud box, may make some sense in gaming and short term. But longer term that seems a bit silly. Look at the iPads benchmark scores - those things are already rediculously fast.