I think the problem is if you have the user affecting the environment, say, it tracks your hands and displays them. You don't just need volumetric information, you need to render the possible actions the user can take over the next few milliseconds.
Or, you split rendering work, and leave some things for the local machine to render.