We're working on something similar - several scene layers packed and transmitted over h265 streams and unpacked into a 3D client for 6DoF playback. Captures from something as simple as a GoPro and then our CV compares perspectives of the scene across time to reconstruct it in 3D for the encoding/transmission steps.
Targeting exercise market (where we got our start) but it could go beyond it in time.
Short demo: https://www.youtube.com/watch?v=DST9jz9Rrcc
Happy to chat, email in profile.