The data for each scene hasn't been streamed to the GPU in a long time, not since the first few generations of GeForce cards. "Vertex buffers" store the geometry in the GPU's RAM, so it's there. The game engine just swaps out buffers as things come in and out of view.

I have the same questions as you regarding how this works with respect to offloading the the scene graph data structures though.