At the end of the day, I couldn't get it to work despite having some great data sets. I talked to someone who did manage to get this work, proved down to centimeter accuracy but they were only able to get it to work with great difficulty and a rig with 5 different cameras. They said by the time you do all that, you'll have invested enough that lidar would make as much sense cost-wise.
I believe the problem is at the MVS step, but it seems like none of the libraries handle this use case very well for some reason, despite having great overlapping features.
When taken on the ground, doing orbits looking in toward a common object works reasonably well, and colmap handled this the best. Especially with exhaustive matching, but it takes DAYS to process. (Or, 4-6 hours if you REALLY optimize the process.)
But moving in a line, no matter how much overlap of the features you get, just doesn't work that well, and I do not understand why. I think it fails in the feature matching step and it makes no sense since it has really similar images to try and match if you capture at any reasonable frequency.
I think THIS worked better than I would have expected because of the 3d aspect where the lens gives you multiple perspectives on objects along the sides.