Hacker News Reader

Comment by wpietri | Hacker News Reader

This is a really sharp approach, where synthetic scenes are rendered and used to train a system so it can better understand real scenes. It reminds me of the bit with Deep Thought in HHGTG, where "even before the data banks had been connected up it had started from 'I think therefore I am' and got as far as the existence of rice pudding and income tax before anyone managed to turn it off." This isn't quite that, but it's a good step in that direction.

View on HN

This could be used that way, and there is some precedent, e.g. in RenderGAN (https://arxiv.org/abs/1611.01331). Note the GAN part: that's why the "differentiable" part is important, so you can integrate rendering into a trainable network. Otherwise, you don't really need the "differentiable" part, and in fact using synthetic renderings from traditional, established renderers (both rasterizing and raytracing) has been a pretty active area for a while now.

There's also some approaches like SUGAN (https://machinelearning.apple.com/2017/07/07/GAN.html), but those don't really use differentiable rendering -- they just use traditional rendering, then apply a 2D GAN on top to make them look more realistic (from a CNN's point of view, at least).

To me it sounded really similar to a Generative Adversarial Network (a GAN). With a GAN you have one network whose job is to classify an image (is this picture really a person?) and another whose job is to essentially fool the classifier (generate an image that looks like a person).

This case is a little bit of the reverse, in that it's focused on making the computer vision component (the discriminator) try to match the visual content that has already been generated.

Seems like these types of "adversarial" approaches will be used in lots of different domains, as so far they've produced some pretty amazing results.

Speaking of doing things the other way around, could ML techniques be useful for tuning shader parameters and light placements in order to make a 3d scene modeled by a human look as close as possible to a reference photo?

(If so, it should probably be done with multiple reference photos from different angles, to ensure shaders and lights aren’t adjusted badly so that the scene only looks good from the one angle that the computer was looking from when it was tweaking.)

Sure, I can imagine that. Would be surprised if there isn't SIGGRAPH papers to that effect. At least initially, it would be more of an optimization problem, but ML could help as well.

Would be even nicer if it could be trained on unpaired datasets (ala CycleGAN https://arxiv.org/abs/1703.10593).

Reply on news.ycombinator.com