I assume the real challenge here is preventing some kind of "pop" when a pixel leaves or enters the eye's area of focus. Any kind of change in the periphery of your vision would be distracting.
(brains "drops frames" while the eyes move, so "popping" during eye movement probably won't be perceptible)