Where does most of the delay come from? Is there a way to take a "cockroach" nervous system approach where certain sensors have as direct a connection as possible to the code that updates the view of the user?
You might be interested in the articles Carmack[1] and Michael Abrash[2] have written about VR.