Jittery rendering, hangs, and high-latency are deal breakers in Virtual Reality. I think things will get better as a matter of necessity for VR and AR, and then "veterans" from the field will bring back "new ideas" about performance and user experience.

Or, if you prefer your optimism a bit more dystopian-flavored, some megacorp will come around with a walled garden whose user experience is just so good that users will flock to use it, and the rest of the industry will have to adapt to compete.

In either case, I don't think getting used to it is our only choice :)