I think 60 frames per second would be smooth enough. Walking speed 5km/hour is roughly 1 meter/second. Per frame distance then is about 0.02 meters.
Looking at the map the YC office seems to be about 30 x 40 meters. Imagining a grid with lines each 0.02 meters overlaid on it, it seems you would need 3000000 panoramas. That might sound way too many, but wait! We live in the future.
One spherical panorama of 10000x5000 pixels seems to be about 4MB jpeg-compressed. So you would need only 12 terabytes of space. Also since you need 60 of these each second, you need storage that can move 240MB/sec, which is lower than the current speeds of SSDs.
1TB SSD seems to cost about $400, so for only $4800 you would have enough speedy space to store the panoramas. Enough space to explore whole YC building with no snapping at all between frames, with complete realism. Actually you could even do stereo 3D, as you already have the data.
Now it's a different question entirely on how we could take those 3000000 panoramas. Even if you had a Double Robotics bot with a spherical camera attached to it going around the space, snapping 10 panoramas per second, it would take 4 days to complete. While that itself is tolerable, how it would know its position and control its movement to the required accuracy I have no idea.
Still it blows my mind that storing all that would be possible.