Ask HN: What's the algorithm behind Polycam 3D reconstruction? Any one?
Anyone who can reverse engineer and tell the algorithm behind polycam app? https://poly.cam/
It is just amazing!!
It is just amazing!!
It seems they have multiple operating modes. One mode use rgb image only and standard photogrammetry like meshroom. In the other mode they use the information from the LIDAR sensor to help with depth and camera registration, with some sort of RGB-D SLAM algorithms.
Looking for point-cloud fusion algorithms and the pcl library should get you started.
There is probably some deep-learning involved but it's not yet end-to-end deep learning, in particular this is not NERFies.
If you look at the recent publications of their Senior computer vision engineer, https://scholar.google.com/citations?view_op=list_works&hl=e... you will find the following "A Learned Stereo Depth System for Robotic Manipulation in Homes" https://arxiv.org/pdf/2109.11644.pdf it should get you some approximate idea of how it work.