What's the state of the art for structure from motion? Given video of a space, how does one practically turn it into a 3d scene today?
As far as I know, the basic approach to SfM hasn't changed much in the last decade or so. It boils down to image feature extraction (using something like SIFT feature vectors), then a heuristic matching process, then bundle adjustment and outlier rejection.
Is there something off the shelf that does this, or a library that can generate these point clouds so I can go about trying to fit simple geometry to them to reconstruct the scene?
Please forgive the noob questions; I've not worked with automated 3d stuff or photogrammetry before.
It's a pretty decent tool, although not sure if videos work well.
I once tried to build a mesh from an aerial video by extracting certain frames and the result was ok-ish
What does this mean? Why would scale matter in a 3D model when you can scale it however you want in two seconds?
Also they said 'to-scale', but none of this applies, since if you are doing some sort of photogrametry you are getting an accurate model of what you are photographing anyway, that's the entire point.
You can supplement images with lidar, stereo images, or IMU odometry to capture true scale information at the SfM stage to get (approximately) scale-correct models.
Alternatively you can learn what the real world tends to look like and estimate absolute depth from images alone, if you have enough data. This might be defeated by toy cows and doll houses, but work for some applications.
This is not a term that makes sense. It makes a model, there is no sense in having it scaled in any way except for being correct. Saying 'up to scale' is like saying 'feet running' or 'mouth eating' or 'face talking'.
the scale is arbitrary
Only if you don't measure anything. If you have the camera correct and know how high something is or just know some depth or distances photogrametry makes an accurate model. 'Scale model' is a term from when scaling the model down made it easier to make. In the computer, it just means it isn't accurate.
It's used when you don't know the scale.
You can estimate all the geometry 'up to' but not including a scale term.
> and know how high something is or just know some depth or distances
There you go: you just added in some extra information to constrain the scale estimate as I described.
I recently tested exactly this use case, with iPhone apps 3D Scanner, Polycam and magicplan.
The results were, how do I put it nicely, not very useful.
For example, the floor wasn't even flat/straight! One would think this would be a basic constraint (or “inductive bias”) of the algorithm…
I haven’t tested Luma yet; also my example wasn't literally the most basic one (square empty room), instead there was some clutter on the floor, irregularly shaped room, a chair in the middle of the room, and open doors.
(Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?)
Can I ask why you say this? Camera motion models including Kalman filters of various kinds and constraints on derivatives etc, are absolutely used in SLAM and photogrammetry afaik, and have been for a long time.
This was a couple of years ago probably. Meshroom and Reality Capture. Maybe I was mistaken or maybe they've improved?
Can you point me to software that does use video for photogrammetry in the way you describe?