You're getting very close to the motivation behind VisionOS, aka a 3D spatially oriented operating system.