Since most VR experiences are immersive and take up your entire view, you barely see the OS other than first time setting it up and then launching apps.

Would be nice if it was better, I don’t see it as a “huge downside” as you do though.