Ok so I've tried doing some version of this that is a bit more advanced [0] but I gave up because I'm not a ML expert. Have you thought about creating/projecting video versions of lightfields? Like Google's Deepview [1]. I'd love for DeepView Video kind of tech to be comoditized.