My concern is primarily power draw. For a device with only a 2 hour battery, every erg matters.
Go to an AR web page on your phone and start playing around with 3D objects in your space and you'll notice your phone getting significantly warmer and drawing more power. The Vision Pro is doing this literally all the time.
Also, the camera and sensor work that is tracking your eyes has to happen whether or not there is an outward-facing display.
Apple makes a watch with a display that is always on, and their phones have high resolution OLEDs that can stay on for over 13 hours (iPhone 14 Pro Max 150 nits brightness doing continuous 5G web browsing).