I wonder if some of the latency could be soaked up by a physical adaptation of the display panel. Imagine a display that adjusts position manually/mechanically in relation to the head movement, enough to give the latency time to catch up with the underlying movement. People won't just sit in a chair and spin 360 constantly. If the physical display could shift the angle of view or the pan/scan of viewable pixels perhaps the rendering could happen a bit slower and catch up when the user stops for a second to focus etc.?

You could render a 'latency' buffer past the edges of the panel's physically viewable viewport and then the panel could expose that zone along with head movement while the rendering continues to work on catching up with the actual shift. When the head comes to a stop for even a handful of milliseconds the display can recenter while the scene is rendered to "time current" position.