Right, I would assume we'd render multiple 2D images from different angles, and train on that.
But as I'm writing this, why can't we do ML directly with the 3D models...
But as I'm writing this, why can't we do ML directly with the 3D models...
If you can figure out exactly why that is, you will at the very least get a very well-cited paper out of it, if not win some sort of award. It's not a complete mystery what the problem is but the solution is unknown.
Because we don't know what the difference is, we can't fix up the 3D renders. "Just make it noisy" certainly isn't it. (We in fact have a lot of experience throwing noise at things; the whole "stable diffusion" AI image generation is based on that principle at its core.) It has to be the right sort of noise, or distortions, to represent the real world, and nobody can tell you exactly what that is.
Yes, you can train an ML model on rendered data, but the model tends to fixate on rendering artifacts, and performance doesn't transfer to real world images. Plus, it's very difficult to make generated scenes with the variety and complexity of the real world. Your trained model will fail to generalize to all the distractions in natural scenes.
Yes, there are techniques for all these problems, but none of them are good, reliable, or easy to get right.
(DL models are trained with one type of camera and used with another type of camera all the time, and that's sort of similar; plus renderers as used in blender3d are pretty good and they should work well with LEGO bricks which are relatively simple objects; if despite all this it doesn't work you could degrade both the real and the generated images e.g. by quantization, with the aim of bringing them closer together.)