Filament: Physically-based rendering engine
google.github.io
google.github.io
http://www.curious-creature.com/2017/08/14/physically-based-...
Good example of visual state-of-the-art on Android is probably the ARK dinosaur game. The Vulkan API is now included on latest Android. And with the compact gltf 2.0 format we can expect great experiences on mobile ;)
Vulkan glTF 2.0
https://www.youtube.com/watch?v=sl7iN-vQCOs&list=PLy80eMh1-z...
I looked at the COLLADA and glTF formats (I wanted something open) and very quickly chose glTF 2.
I don't have any 3D format implementation experience, and I implemented glTF 2 in Golang in a day, or less.
Anyway, it's a great format (AFAIK) and I wish more 3D modeling tools supported it. I imagine more will come on board as time passes.
(I'm not agreeing or disagreeing with them, I'm genuinely curious to hear from someone working with the format at the implementation level)
Scene structure in v2 offers everything Creo View offers and pretty closely matches what is available in the Unity3D hierarchy, from what I've seen. You assign one or more meshes to a node element (rinse and repeat) and you arrange them in parent & child configurations. Transform matrices of parents pass down to children, and so on. If a child node has its own matrix specified, it is multiplied with the computed parent matrix, resulting in the final transform applied to whatever mesh(es) defined in that node. Exactly the same as what Unity or Unreal do.
So the scene structure criticism doesn't hold water for me.
Same for materials. GlTF v2 supports PBR materials with or without texture maps, normal maps, occlusion maps, and emissive maps. All of those are optional and you can set things like color and normals, opacity, and emissivity all without textures if you want.
Most glTF tools out there, including a lot of open source things like blender importers and exporters, assume that there will always be textures to define those material properties, and completely ignore non-texture material parameters like BaseColorFactor, which is incredibly annoying because NONE of the models at my work use textures for anything.
Things like lettering on placards is modeled, not textured, and no UV maps exist for anything. This will be an interesting/infuriating problem when it comes time to view these models in a game engine in a production setting. Optimization in game engines relies on things like textures and UV maps.
To that end I have a to-do item on my list to simply generate small map images for these things and set give each material one pixel on the maps and make all UV coords for a mesh with a given material point to that pixel. That will probably work, and it will probably appease the half-implemented open source stuff I've come across. Runtime lightmaps will not work this way, however.
I can't speak to the animation features of glTF, since I'm dealing with CAD data. Our models are individual parts assembled into whole assemblies or vehicles at runtime, and not stored as whole models.
I think for lettering and similar things (high-complexity but essentially flat) you're right. You could probably use quads here to give you easy UV mapping and script the creation of textures.
What else have you modelled which would ideally be done via textures in a real-time engine?
A full fidelity CAD model for something like a bolt can easily be two million polygons if the draftsman isn't careful.
You can get 80-90% there but miss quite a bit of fine detail needed as a interchange format (which is why this is a transmission format I guess). Extensions seem to be basic and I question how well they work in 3d editors but not really sure how that works and is probably editor specific. I've had a lot of issues trying to have a single file like collada be properly supported in multiple editors unless it was fairly basic model. That leads to a lot of wasted time for modelers trying to get the edited model back into the game and working correctly.
Back to GITF, collision seems to be missing. They admit that they dont have vertex compression implemented or animation for things like material colors or animation metadata (timelines, looping, ...) . Is there a way to describe a partial surface sort of like skinning for other reasons (for example Fallout 4 needs to tag certain surfaces/vertices as to say that they are part of the head or arm for decapitation in game). Anyway looks like a nice start as its all JSON (or binary) unlike GEX,obj,3ds,... which seems to need its own parsers to read and they did only claim to be an transmission format. Again only did cursory read of the format so I might have missed things.
It's understandable. Before glTF, there really wasn't a good non-secret/proprietary format with which to deliver 3D content. But, now there is! Use it!
GlTF is a runtime transfer format and that's mostly how I'm going to use it. Others at my employer need to do VR work, and sometimes that requires the geometry to make a trip through something like 3ds Max. It would be nice if Autodesk acknowledged the existence and utility of glTF. People want to move away from FBX so Autodesk are unlikely to provide glTF support until it is demanded of them by a large enough portion of their audience.
Once I am done with this Creo View stuff I will look at writing a 3ds Max import plugin myself. It would definitely get used by my employer.
Writing an importer for those ASCII files is easy, and when you have the binary and ASCII versions of the same files side-by-side, working out the binary file format is made much easier.
I haven't even begun work on a 3ds Max importer.
https://vulkan.gpuinfo.org/vulkansupport.php#android_devices
Is there a PDF version?
The document uses a JS tool called MarkDeep to convert extended MarkDown into styled HTML. Just look at the source, it's 99% plaintext Markdown.
If you read the MarkDeep docs, you'll find that it has a feature to convert the source into PDF instead. I admit I haven't tried this, though.
If it were up to me, I'd do the transform offline and just serve the static HTML instead. And sure, a link to the PDF just to be nice :)
Why is it still so hard to design a page to load reliably?
Additional docs on their material system:
https://google.github.io/filament//Materials.md.html
https://google.github.io/filament/Material%20Properties.pdf
Also, I'll plug http://casual-effects.com/markdeep/ which was apparently used to format the docs.
The renderer is perceptually better than the one that is included in my CAD package (TurboCAD) for pretty much all materials. So I'm guessing they will snarf it and dump the proprietary renderer and replace it with this stuff if they can.
But the really interesting idea that popped into my head was this; could Google offer 'rendering as a service'?
Specifically they have a zillion machines, many of which are doing nothing important, and they have this rendering package, and they have a scheduler that can put things on any machine. Imagine a service where you sent them a suitably detailed model description, and a set of lights, could they send you back a rendered image? Could you parameterize changes to the model description over time so that they could send you images in time based on your models? Could they do say 480 renders 'free' per month and then maybe $0.19/render over 480 in a single month?
Could you create a studio of modellers who would design models, and animators that would animate those models over time, and a director who would compose those animations into scenes? This is basically Pixar without the expensive renderfarm. Does that enable new studios to bring their own vision to life? Does it offer a cost effective service to places like Pixar which allows Google to make money on otherwise idle resources? Curious minds want to know :-)
Here's a session from GCP Next 2018 for cloud render farms: https://www.youtube.com/watch?v=ODOJ3UbnV6Y
[0] https://web.archive.org/web/20171017063005/http://home.lagoa...
However I have no knowledge in the industry, just a gut feeling
IIRC the issue is that if you can ignore fluorescence, then reflection is simply an element-wise multiplication of the incoming light at the wavelengths under consideration[1] with the reflection coefficient of the material at those wavelengths. With fluorescence, that turns into a matrix multiplication, with obvious speed implications.
If only a single wavelength is considered at a time, then the wavelength must change upon reflection, otherwise there's no way for the fluorescence to occur. That can also have performance issues, for example conversion coefficients to/from regular color spaces needs to be recalculated.
At least that's my understanding having worked on a physically-based renderer which did do spectral rendering but not fluorescence.
[1]: using for example binned wavelengths or stratified wavelength clustering.
(I'm reminded of some company that developed a pair of glasses for color-blind people which enabled them to see "color". I think it was a similar princple.)
- colors that were previously the same could be now perceived different (a red car, a red flower, red light from LCD screen etc...)
- White / grey would still exist but be less common, as a lot of white / grey light would now be close to a new receptor
- Probably more distinguishable shades in the rainbow
Granted, our trichromatic vision is likely adapted to the scenery of this earth. We are most senstive to hues of green for instance. That being said, all things equal, wouldn't new grays appears with the new cone sensitivies just as the old grays would now yield a color response?
>- Probably more distinguishable shades in the rainbow
What I'm curious about is those pure hues we perceive that are infact the result of composites of wavelengths. Those equalities would break up. Two things that were once, say, equally orange could suddenly have different hues in the new mapping.
I'm even wondering if we all "see" / "perceive" the same colors / have the same response to the same hue (i.e. is my blue the same as yours ?!)
wouldn't new grays appears
Well, the prism/rainbow is the pure wavelength specturm, so that's one of the things that probably would turn out the same in the new mapping.
>I'm even wondering if we all "see" / "perceive" the same colors / have the same response to the same hue (i.e. is my blue the same as yours ?!)
The sensitivity range does vary between individuals. Some women are in effect tetrachromats because they have two sets of, I think it is, Red cones that are sufficiently wide apart. I find this very fascinating.
The downside is that this is slow. An RGB-ray traces the whole spectrum for one ray (approximated by an RGB triplet). The single frequency ray does not. If you try to send a binned spectrum in the ray, what will happen is that as soon as you hit something refractive, they will go in different directions, so you no longer have a full spectrum. All the complexity in a spectral renderer over an RGB one lies in handling performance: How to avoid the performance problems when you lose coherence. Apart from that it’s actually often simpler than an RGB renderer despite being able handle all those fancy spectral effects.
For a nice “reference” or “tutorial” one that isn’t plagued by the performance hacks needed to make it viable for large scenes, look at this one: https://github.com/TomCrypto/Lambda
Ironic aside: Notice the "Processing math" message on the lower left when you first open the page? That's MathJax, a huge JS library, chugging away because Chrome doesn't support MathML. Try going to the page in Firefox and you'll see that it processes "the math" a hell of a lot faster than Chrome. Firefox supports MathML, and MathJax probably generates MathML for it.
2. How does it compare with what OTOY are working on for realtime? I imagine they are focused on quality over performance and aren't even considering mobile.
Here’s a bit of convo about the code and art of R&C
https://youtube.com/watch?v=Y65h1aO-xps
CC=clang CXX=clang++ ./build.sh -j release
to get it started to compile (as it seems it had cc pointing to gcc and it did not understand some, apparently clang-specific flags).
Here’s food for thought: what does it mean to multiply two colors? Nothing. It’s meaningless. It’s an approximation that happens to look good. But if you’re going to claim your engine is physically based, you can’t use it, because multiplying two colors is not based in physics.
https://en.wikipedia.org/wiki/Physically_based_rendering
https://marmoset.co/posts/basic-theory-of-physically-based-r...
Similarly, what does it mean to add colors? Well, kind of non sequitur. Depends entirely on what abstractions you are working on.
The claim is precise: It's meaningless to multiply two colors, yet this is the core abstraction that we use in our rendering engines.
New ideas are needed to advance the state of the art.
In fact, multiplying two RGB colors is about as meaningful as multiplying the color of an apple by the color of a dog. It's certainly not rooted in physics.
Those bands are arbitrary, defined by human color receptors. If we were dogs, would we be arguing that lighting should be defined as {a,b} * {x,y} = {ax,by}? Dogs can only see blue and yellow. So what's special about r,g,b, and why do we mutiply them together?
There's nothing special about them, and it's arbitrary and misleading.
>Those bands are arbitrary, defined by human color receptors.
Sure. And if we had monochrome vision then we could have used just scalars for light. If you are arguing against 3 component color model then you should not have been baselessly attacking such uncontroversial thing as attenuation. There are engines, which use more points to represent spectra yet they still use multiply.
And how could it be otherwise? Real life looks correct to any creature that looks at it. And our renderers supposedly model real life.
This is a contradiction, and rather than trying to disprove what I'm saying, it's worth chasing down the logical fallacy.
I am not discussing "correctness" whatever it means, I am arguing against your assertion that "multiplying colors makes no sense". Are we done with that?
In that context, it makes no sense to multiply two colors together.
Impossible. RGB is just a 3-bin spectrum. It's the minimum number of bins needed to represent any color humans can perceive because we have 3 color sensors that are at least partly independent.
You could always imagine a hypothetical creature with yet more sensors, until you need a bin for every possible wavelength of light. Eventually you'd effectively be simulating individual photons. The world has too many atoms for us to simulate it at that level.
We sometimes use larger spectrums than RGB because while humans may not always be able to perceive the difference between two different spectrums of equivalent color, sometimes they interact differently with other things.
For example, white light formed by the full spectrum of visible light will seperate into a rainbow when passed through a prism. White light formed by red, green and blue lasers may be indistinguishable when viewed directly, but will only seperate into red, green and blue when passed through a prism.
The number of bins we use for that sort of simulation depends on how many we need to get a result that is a satisfactory approximation of reality. All models are wrong. The question is always whether they are good enough for our purposes.
Some can, but it’s obviously much more computationally intensive. Outside of some niches, it’s also not particularly useful because the final product will need to be in RGB anyway if it’s to be displayed on a monitor or TV. Even if you were to directly inject the renderer’s output into the retina, there are lots of....wavelength distributions that will produce identical percepts, so it’s not a clear win even there.
But you do get a fair bit more spectral noise in the image, but this can be handled by handling multiple wavelengths at once with SIMD and importance-sampling the wavelengths to use (Hero wavelength sampling).
Where it does actually make a lot of sense over RGB triplets is for volume scattering / absorption (so more accurate hair and skin rendering for example) - calculating the mean free path with wavelengths is much more accurate than using an RGB triplet which is an approximation - however, this means to enjoy this benefit, you need to store the volume coefficients in wavelength bins (instead of RGB values), which uses quite a bit more memory.
Simply that our monitors reproduce images using red, green and blue light.
A PBR engine models the entire process of the light from a scene, falling into a lens, measuring the light that hits the imaginary rectangles that are the pixels of the output image.
A PBR engine can model the light internally as a continuous spectrum using Monte Carlo, integrating paths of weighted-random wavelengths at each pixel.
You could stop here if you just wanted physical accuracy, you got the spectral data. But if you want to see it, you'll have to get it onto a computer monitor somehow.
So, you model a photo-sensitive medium using an exposure function, and apply colour-space transforms to turn the spectrum into just the three RGB wavelengths, in a way that has been measured empirically to be perceptually close to the original spectrum (look up how we got the CIE standards).
This is all science so far.
If you don't care about the full spectrum but only the RGB (or why not CMYK) result, there are some optimisations and approximations you can do because in most cases you don't need to simulate the full spectrum.
Did you ever stop to think that RGB screens might look weird to dogs as well? However, if you somehow built a dog-monitor with only blue and yellow, you could optimize even further. It would be a lot harder to get accurate colour-space data like CIE from dogs, though.
If you wanted to simulate a black-and-white photograph, it would be easier again.
> and why do we multiply them together?
I actually don't think we do, in modern PBR engines.
Multiplying two colours together is done in simpler 3D engines because it models reflection; one RGB triplet represents an approximation of the light spectrum, and the other represents the reflective properties of the material. In every-day language we use the word "colour" to mean both. But they are actually of a different "type", if you will. We say "colour" to both the additive type (light) and subtractive type (material).
What we model is the transformation of our photon path (additive) by the material of the surface it hits (subtractive). In the simpler model this happens to equate an element-wise multiplication (not "just" a multiplication).
However, in modern PBR engines, the reflection operator is a lot more complex than that. The material is no longer simply represented by an RGB triplet, and the operation between the photon packet and the material isn't a multiplication.
You do actually approximate the energy that reaches the different surfaces (in Watt / (m² sr)) for all wavelengths of light using Monte-Carlo integration. You can send probes for different wavelengths separately if you care about the differences in the refraction index of your material by wavelength.
Then you integrate the results by folding it with the corresponding color response of the primaries of sRGB or whatever color space you want to use. The color representation can be accurate, though screens etc are obviously limited.
The closest thing to "multiply two colors" in physics would be taking a product state:
|ψ⟩ = |ψ1⟩ ⊗ |ψ2⟩
Here, we are expanding a Hilbert space to include both unentangled particles (color == photon).
Well, yes. But in this case, the approximations are completely made up. They're not based on physics.
There's a very good reason why people don't solve the kinematics of a tennis ball as if it were a 10^23 quantum-many-body problem. If you're constrained on time and money, the smartest thing to do is to get the closest result you can in the smallest time with the least amount of money.
[0] http://hyperphysics.phy-astr.gsu.edu/hbase/vision/colcon.htm...
I have a screen (it happens to be about 18" wide in this case). Why on earth is the text in a stupidly thin column in a small font? It looks like TeX's daemonic alter thingie got into the render process somewhere or perhaps someone forgot the other two columns or wanted to torture a webby reader with multiple columns (mmmm ArXiv scrolly uppy n downy pdf) and lost interest.
Anyway ... it looks crap in my browser unless I hit CTRL(num)+ a few times - 200% works.
This is what I see:
https://nextcloud.roseandjon.gerdes.co.uk/nextcloud/index.ph... - that should yield two images.
Fonts/typefaces - lovely. Layout - a bit limited.
Nothing, other than the format and experience of downloading a document to be opened with a desktop application.
There's a lot of one click image sharing services out there. I use Windows key+shift+S to take a screenshot and then go to imgur.com which supports reading images from the clipboard, bam, instant URL for my pic.
The brain doesn't require the eyes to hit every single word to extract the full text thanks to peripheral vision, but this optimization is not as effective when the width of the text-line is too large.
I am surprised you didn't mention the FOUC, though. For a while you see raw, minified, un-styled text on this page.