I think you've nailed it here. I never thought of it in those terms before, but now that I see it, it's obvious.
It reminds me of the (probably apocryphal) story of Henry Ford saying that his customers didn't want cars - they wanted faster horses. I think there's something analogous here. If all you are offering is a Linux desktop in VR, you're making faster horses.
When I think about where a novel new VR mode is going to come from, it's either going to be Apple or a research team at a liberal arts college. Technologists can really only solve half the problem.