Hmm on second thought you could also just render the "overscan" and have the IMU work with the display to shift the pixels into view while the processor works to keep the overscan updated. On pause you can re-render everything (overscan included) and start all over again on the next movement.
+------------------------------+
| <<< "over scan area" >>> |
| +----------------------+ |
| | viewable by user | |
| | | |
| | | |
| | | |
| | | |
| +----------------------+ |
| <<< "over scan area" >>> |
+------------------------------+
This might leave enough "headroom" too keep up with potential movements at very low latencies.