And apparently the effect is much more jarring in VR - which is why Valve solved it for Half Life: Alyx

https://www.youtube.com/watch?v=h_MuWmYKGIc