There also seem to be a lot of anecdotal suggestions that learning how to use that AR/HUD single-eyepiece thing when flying the Apache helicopter similarly requires some brain-reorganizing shenanigans to fully master.
[1] Somewhat amused, the instructor pilot let it happen for a few seconds before asking me if I was OK and could see what was happening. It was a formative learning experience.
I think the formative learning experience is why the instructor let you drift for a while. Of course, plus 20 feet has a different impact on safety than minus 20 feet :P