Hacker News Reader

Comment by simsla | Hacker News Reader

> plus data on the camera angles they were taken from

Doesn't seem like much of a stretch to determine the angles as well.

E.g. a semi brute forced way with GANs

I've spent a lot of time thinking about this (i.e. taking a video and creating a 3D scene) and I don't think that it is feasible in most cases to have good accuracy. If you need to infer the angle, you need make a lot of biased assumptions about things like velocity, position, etc., of the camera and even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time. Now I'm not saying it's not possible, but I'd believe that if you want an accurate 3D scene, you'd rather be spending your computation budget on things other than determining those angles when it can be simply be provided by hardware.

https://github.com/NVLabs/instant-ngp has a script that converts a video into frames and then uses COLMAP ([1]) to compute camera poses. You can then train a NeRF model within a few seconds.

It all works pretty well. Trying it on your own video is pretty straightforward.

1. https://colmap.github.io/

You're far too pessimistic (or maybe you don't know the field well). The problem of estimating the relative poses of the cameras responsible for a set of photos is a long standing and essentially "solved" problem in computer vision. I say "solved" because there is still active research (increasing accuracy, faster, more robust, etc.) but there are decades-old, well known techniques that any dedicated programmer could implement in a week.

If you're genuinely curious, look into structure from motion, visual odometry, or SLAM.

> even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time

Not really, with SLAM there are various algorithms to keep inaccuracy in check. Basically it works by a feedback loop of guessing an estimate for position and then updating it using landmarks.

You don't even need anything that fancy. Traditional structure-from-motion, or visual odometry gives accurate enough position estimations.

If you want to experiment, take a bunch (~100) of photos of an object, and use COLMAP to generate the poses. COLMAP implements a global SfM technique, so it will be very accurate but very slow.

Reply on news.ycombinator.com