Comment by glinkot | Hacker News Reader

Comment by glinkot | Hacker News Reader

This is a really worthwhile goal. I grabbed my Quest 2 and jumped onto the site. I wondered if the artifacts around areas in motion (eg the parachutist) are from compression or the 6DOF conversion process? Some 'tiling' effects on the grass areas had me wondering the same, but I imagined these were because the algorithm generates a certain size mesh even when the surface is fairly flat.

What options are there for filling in 'unknown' regions? On the fire spinner video, you can move your head from side to side, but you get darkish blobs shadowing out from behind the performer and the trees. That got me thinking about stereo separation with multiple 180 vr cameras as you mentioned, and how many 360 degree cameras (and/or TOF sensors etc) we'd need to approach real time photogrammetry and a scene where the VR user could walk a meaningful distance within the filmed environment. How plausible would that be given all your experiences in the area?

I'd love to see some downloadable full quality ones to see them in all their glory! Best wishes with it.

Artifacts at the edges are due to occlusions. An occlusion is a part of the scene which wasn't visible to the original camera. You see these if you move far from the original camera in VR to look behind something. This is a really hard problem for 6DOF. We've been improving the quality of occlusions over time, e.g.: * v1: https://lifecastvr.com/demo_maui.html * v2: https://lifecastvr.com/kalalea_fire.html * v3: https://lifecastvr.com/hubner4.html Version 3 now uses a 2-layer representation which has an image+depthmap for the background layer, which is drawn to fill in the occlusions. This background layer can be precomputed in a variety of different ways. For example, here is a CGI synthetic scene where we can construct the background layer perfectly: https://lifecastvr.com/liferay.html However, making up the background layer for real-world content is more challenging. We are on version 1 of that. We will improve this with machine learning in a future release. We can also substitute a "plate" 3d scene for the background in cases where the camera doesn't move. We have also experimented with using data from other frames when the camera moves. This will improve over time.

When moving onto multiple (depth) camera setups, in-painting from old frames worked really well, even before any masking off of static vs moving content (done in realtime, for live streaming)

Reply on news.ycombinator.com