I think it might be a bit like the hollow face illusion:

https://en.wikipedia.org/wiki/Hollow-Face_illusion

When I was playing an early version of HL2 VR if you walk right up to the wall, the crowbar you are holding goes through it but the wall stays at the same distance. A very weird feeling.