Its worse than that. Cameras (plus DToF if you want tracking in the dark), imus, gyros, and necessarily onboard compute/SOC to handle processing that data. Shipping it all off to a remote computer and then making the round trip creates an untenable amount of lag. Thats not even accounting for controller and hand tracking.
And once you have the pipeline and computation power to enable inside out tracking all on device, adding an OS is essentially free.