You need to look right at where they want you to look - so in this example you need to look right at the center point of the image. If your gave drifts off to the side then it looks bad.
The same with 3D movies, if you don't want to look directly at what the director wanted you to look at, you end up getting seasick.
Do VR headsets have gaze tracking these days?