Isn’t that an intractable problem without LiDAR? I suppose the naive solution would be to fix the screens in spots as you suggest as opposed to pinning onto real life locations.

edit: ah, right the vision pro does have lidar. I look forward to seeing this pinning if it's implemented. I'm still skeptical it would work well - I've used ARKit and it's not really that accurate when you're moving around.