Mitsuba 3 Physically Based Renderer
mitsuba-renderer.org
mitsuba-renderer.org
That writeup is PhD Thesis material in terms of scope and detail, but the student that wrote it, Tizian Zeltner, actually wrote his PhD on another topic entirely. This is just the level of stuff that routinely comes out of that lab.
Make no mistake, this is a very high achievement, and is very rare in both private companies and research labs. It's a very hard point to reach if you don't like or care about what you are doing.
Kudos to them.
https://shuangz.com/courses/pbdr-course-sg20/
Inverse rendering is such a cool idea... solving for the inputs (3D models and materials) to a renderer such that they match real world inputs (photographs) by computing gradients automatically.
Most differentiable renderers stop at the object surface, but differentiable volumetric light transport is what would allow us to use the renderer to optimise a measurement device or for medical diagnostics or just to estimate the optical properties of a material (that is absorption and scattering coefficent and scattering phase function), which is a pretty hard problem in itself.
From a quick look, Mitsuba 3 seems to have everything in place to be used in such a setting and I'm really excited to see some use cases outside of Computer graphics.
Mitsuba – A physically based renderer - https://news.ycombinator.com/item?id=10096737 - Aug 2015 (8 comments)
See the final results at https://mitsuba.readthedocs.io/en/latest/src/inverse_renderi... (scroll down a bit to see the reconstructed image vs the original)
No longer will you need a team of 100 artists for every 2-3 developers to manually model and texture objects, instead you'll have 10-20 photographers running around taking pictures of real-life objects and scenes. These will then be reverse-rendered into models, which can then be directly placed into the game world by just a handful of map designer artists.
You can see this in Unreal Engine 5, where they combined their "infinite detail" polygon engine Nanite with their acquisition of Quixel's Megascans library of models captured via photogrammetry.
However, photogrammetry is not the same as reverse rendering: it's only really suitable for diffuse objects captured in good conditions (foggy or overcast days). It can't handle transparency, reflections, or complex/dynamic self-shadowing. Currently it's mainly used for backdrops like rocks, logs, and road surfaces.
Reverse rendering can in principle capture anything a renderer can do, including specular materials. It can "undo" self-shadowing and produce objects that can be re-lit in different scenes.
For gamers, these are going to be exciting times!
For graphic artists... perhaps also exciting, but not necessarily in a good way.
But there are already commercial and free libraries of commonplace items like furniture, cars, and buildings. So isn't what I described pretty close to the same situation today already?
I think you're missing an important aspect of art - style.
The CG industry has been chasing photorealism for decades and we've been slowly inching closer. Photogrammetry is an important step that will provide a big boost in that goal.
But what happens when we get there? Once it's (relatively) cheap to make a AAA game that's virtually indistinguishable from reality, what happens next?
Everyone will love it for a few years but then they'll start to demand something new, and that might be non-photorealistic styles. Or it might be off world or impossible environments. Photogrammetry won't help much with a game set in space in the distant future, populated with non Earth creatures, plants, and architecture. Although it will help a bit, of course. There will still be a place for ultra realistic environments (war games, spy games, romance games etc.) but it will become just one style amongst many.
If anything this might lead to a revival in making real world models and then scanning them. Imagine a Dark Crystal style world except you are the player. All the characters and models were crafted by hand and then scanned into a 3d world where they are indistinguishable from their real world counterparts when viewed in VR/AR or on screen. That would be cool.
As for artists, perhaps they should be worried about AI stealing their jobs. But not photogrammetry, that'll just remove some of the tedious stuff like modeling furniture.
This could provide a new tool, and maybe remove some jobs. But it will make new jobs related to this new tool.
This doesn't diminish the supply of jobs much though, as it lowers barrier to entry, many startups appear to compete, each needing its own artists to feed their AI with new and interesting styles.
And a new category of artists will emergence, one that is expert in specifically crafting styles amenable to AI transfer.
Individuals and small teams will be able to take on projects that historically have required much larger teams of people working over several years.
You are probably way off on your estimates in how this will change the workforce.
Small teams of artists can now produce television VFX that rival what took hundreds of film artists to do decades ago, but if you pay attention to movie credits, the number of artists working on big films seems to be increasing.
Also, it seems to me that AI and procedural generated assets are more likely to be the future than real world scans.
It's odd that that word doesn't appear on http://www.mitsuba-renderer.org/ - unless "retargetable" carries a similar meaning?
So much in computer graphics are just (clever) hacks upon hacks to get something that looks “good enough” but isn’t really simulating physics in any meaningful way (like SSAO, texture baking, bump mapping, etc.). These hacks are much, much faster than simulating the physical process of photons interacting with the world.
I don’t know where “physically based” originated but my first introduction to it was pbrt, which I suspect popularized “physically based” naming.
Differentiable rendering is the name of going from the final image and “reverse rendering” it to reconstruct a 3D model of it.
Differentiable rendering is what it says: differentiating the rendering process. Imagine that x is the scene, then f(x) is the function that renders the image. Then, Differentiable rendering is simply taking the derivatives: f'(x).
Inverse rendering is a process of finding scene parameters x, such that f(x) produces the given image y. This is often achieved by using differentiable rendering together with an optimization algorithm (like SGD or Adam). However, due to the nature of rendering, it's easy to get stuck in a local minimum. Therefore, even a perfect differentiable rendering engine is not sufficient for the inverse rendering.
It's a different workflow that gives you consistency once you know what you're doing, as opposed to the days of Phong shading.
Physically based rendering approaches from the opposite end. It asks, what are the characteristics of the material, in terms of simulated light interacting with a nontrivial surface, so that the thing appears on screen as it should if it were a real object with real light bouncing off its nanostructure surface.
Ray tracing is one method of rendering, but you can physically based render the old way too with fragment shaders. You just won't get global illumination so it won't look as "real".
I do need to point out that many (most?) so called physically based renderers are still RGB based, with all the downsides that come with that. Mitsuba is notable for also having spectral rendering, which is a distinct feature that models light as a spectrum instead of rgb triplet.
This affects the choice of approaches and algorithms, such as using unbiased rendering[3], and their implementation, like using energy-conserving bidirectional scattering distribution functions[4] to describe how light interacts with a surface.
[1]: https://en.wikipedia.org/wiki/Conservation_of_energy
[2]: https://en.wikipedia.org/wiki/Helmholtz_reciprocity
[3]: https://en.wikipedia.org/wiki/Unbiased_rendering
[4]: https://en.wikipedia.org/wiki/Bidirectional_scattering_distr...
I forgot to mention, if the renderer respects Helmholtz reciprocity, then you can chose to either do forward rendering (rays originate at light and bounce until they hit the camera or dissipate) or backwards rendering (rays originate at the camera and bounce until they hit a light or dissipate), or even do both, so-called bidirectional path tracing[1].
[1]: https://pbr-book.org/3ed-2018/Light_Transport_III_Bidirectio...
Am I missing something?
More about it here and why it’s different from (e.g.,) PyTorch https://drjit.readthedocs.io/en/latest/intro.html
It's a research renderer and its key functionality is reversible rendering—going from an image to the scene parameters that would create that image.
The purpose of this kind of renderer is to develop research ideas that will eventually make it into a production renderer someday…
It's almost like I wrote "scene parameters" for a reason.
More info here: https://blog.qarnot.com/an-overview-of-differentiable-render...
> While these techniques came closer and closer to photorealism, another question arose: what if instead of going from a 3D scene to a 2D image (rendering), we went from a 2D image to a 3D scene? As you may imagine, reconstructing 3D scenes from 2D information is quite complex, but there have been many advances in the last few years. This area of study is called inverse graphics.
It relates to GPU ray tracing because the engine can compile either CPU or GPU specific code, specialized to the available hardware and tailored to the specific scene data; makes it so you can render faster than otherwise.
Raytracing tries to create images by simulating how light bounces around an environment. It’s physically-based in that it uses laws of physics/optics to do that (e.g., Snell’s Law when light passes through different media), rather than some method that just happens to look cool.