Yeah the brain is a bit of an extrapolation machine isn't it. Try running on a treadmill for half an hour and quickly stop and see what happens with your vision :)
It seems like the visual cortex is especially quick at adapting to patterns and needs a second to "unlearn" again.