Where does most of the delay come from? Is there a way to take a "cockroach" nervous system approach where certain sensors have as direct a connection as possible to the code that updates the view of the user?
[1] http://www.altdevblogaday.com/2013/02/22/latency-mitigation-...
[2] http://blogs.valvesoftware.com/abrash/latency-the-sine-qua-n...