Differentiable Monte Carlo Ray Tracing Through Edge Sampling | Hacker News Reader

Differentiable Monte Carlo Ray Tracing Through Edge Sampling

people.csail.mit.edu

82 pointsjonbaer17 comments

I've done something similar in beginning of 2017: https://openreview.net/pdf?id=SyEiHNKxx

I had the whole physics engine + renderer differentiable, in Theano. I used it to do control problems though (so using the image to infer which actions to take), not for system identification (i.e. given the image, what are the parameters of the scene). But yeah, I could backpropagate through the renderer too: https://twitter.com/317070/status/821062814798331905

I think more research should go in this direction, backpropagating these physical systems will empower a lot of powerful optimization methods. There is no reason why a physics engine should be treated as a black box. We know the equations and they are differentiable. I'm glad there will now be dedicated code for the gradient of a full blown renderer :-)

I saw this work when you published it. It looks really impressive! Have you continued to work on it? Theano is no longer being developed, so I guess you'd have to switch to something else. What are the next directions to take for this approach?

I tried to submit it to two conferences, twice rejected. Had to leave academia after funding ran out (did get my Phd). Moved the manuscript to a journal and just last week got a really bad review (saying this had been done in the nineties already) and it will probably be rejected again. All of this is not encouraging my old research group to continue in this direction.

So, the paper and code will likely end up in the garbage bin of academia. And Theano quitting was also not exactly helping indeed.

Next directions are exactly what these people do with their ray tracing. The equations of all of these things are rather simple and almost always differentiable. Someone just needs to code them up and start experimenting with what is achievable when you have these exact gradients.

If someone is looking for a technical area for a Phd/startup: such a physics engine with a focus on optimization (so both system identification, system control and system optimization with gradient descent, making all internals of your engine differentiable) is worth millions.

Wow, I am sorry to hear that. I am more interested in differentiable physics than differentiable rendering; it seems promising for training controllers for animated game characters or even real-world robots with transfer learning. I might try my hand at it but I think there is a whole literature on rigid body dynamics that I would have to read up on first.

I would be interested to hear if you have ideas for specific application areas that would be feasible to tackle as a startup. Animated game characters is the best I've got but I'm not sure it's worth millions.

There are more example on the supplementary page: https://people.csail.mit.edu/tzumao/diffrt/supplementary_web...

This is interesting, there has been research work on doing something similar in Monte Carlo particle transport, which is the same idea as ray tracing but with different physics (higher energies, different types of particles, more complex physical interactions like incoherent scatter etc). Still reading, but this article appears to be based on the same general idea albeit with a different objective.

Differentiation of particle transport Monte Carlo codes is focused on different types of tallies and subsequently computing their sensitivities. The inverse rendering problem in 5.3 is actually also broadly applicable to fields like nuclear security and emergency response.

ELI5 please

Rendering a scene is a function that takes lights, camera, geometry, textures as inputs and outputs an image.

This function is very complex and hard to invert. But, and this is the interesting thing I got from the article, it is similar enough to deep learning functions that the same strategy works: We can take gradients of it and optimize the inputs to fit the target output.

So, while we can't compute the 3D model that would generate the target picture directly. We can ask it "How do we nudge the model to make it more similar to what we want?". If we keep nudging it, eventually it will be very similar. The tool will find the exact position of the lights, the textures, model, etc.

What is the purpose of doing this though? Why would you want to find the original parameters of a scene that was ray traced? Wouldn't those parameters already be available?

The potential uses I see are: 1) Given a finished 3D render and a first pass at recreating the scene, nudge your scene closer to the render 2) Given a photograph and a first pass at recreating the photograph, make your render more photorealistic.

No,you cant tell how pixels will change when you nudge a camera normally without recalculating everything. If environment is differentiable you can cheat and see for instance that you needed to move your arm a little to the left last bowl to get a strike (moving your arm effects your hand which shifts the ball which knocks down more pins which increases your score)

Given an image from a physical camera, convert it into a 3D map with comparable properties.

A differentiable renderer can efficiently answer the question "If I tweaked this input parameter (e.g. camera position, object color, light direction) then which output pixels on screen would change, and by how much?" With this information you can run an optimization algorithm to repeatedly tweak the input parameters in the right direction to make the output pixels look more like a target that you want, e.g. a photograph.

Even better, using the chain rule you can connect the differentiable renderer to other differentiable things, like a neural network controller for a simulated robot. Then you can ask e.g. "If I tweaked the robot's arm position, which camera pixels would change?" This would allow you to, given an image of a target configuration (e.g. two stacked blocks), optimize the neural network to produce that image by controlling the robot (e.g. by stacking the blocks).

I think it's doing this: https://en.wikipedia.org/wiki/Gradient-domain_image_processi...?

Don't think so. This solves problems in the 2D domain, but the paper solves problems in the 3D domain.

It's not really about image processing, rather scene processing. A scene is kind of like a parameterizable hyper-image that depends on real world parameters (such as angle, light diffusivity, etc.). OTOH not all images come from scenes.

This is PyTorch compatible, which is dope.

Reply on news.ycombinator.com