Our visual system is optimized, rather extremely, for understanding 3d scenes under the simple perspective model that our eyes are based on: x' = (x * f) / z
Outside of that 99.999% experience norm, that are brains are so used to, is disconnect and discomfort. If you've ever put on a new pair of glasses, with a different prescription, you'll understand exactly what he's talking about: depth offset and dizziness.
The disconnect is why refraction and lensing is interesting to look at: the model your eyes are used to seeing, for the world behind the thing, is not normal.