>imagine whole movies shot in this format. The fly-on-the-wall fantasy of movies, without being locked in place for the whole shot.
That would be terrible. Framing is the major expressive feature in cinematography, and any interactive format needs a lot more thought put into it than just having a free camera. Literally the worst of both worlds.
Light field video streaming is a thing, however it's pretty niche. OTOY pioneered holographic codecs and light field videos with some degree of freedom and a sense of depth more than a decade ago.