Nitpick: The physics of light is understood. Human vision is also understood. But critically, human vision is completely overlooked in most graphical applications. It's not accurate to say that "the path toward making a computer-generated video indistinguishable from a camcorder recording is just a matter of sampling better." There will need to be a fundamental shift in the industry's thinking before realism will be achieved.
It's a fascinating topic that occupied many years of my life, though this is a bit off-topic.
Thanks :)
It was my focus for a long time to achieve perfectly photorealistic rendering. I come from a gamedev background, and I've been fascinated since around age ten about how to get a computer to paint pictures. By 17 I was writing game engines mostly equivalent to quake, and used this portfolio to get into the industry. The next three years were spent getting as close as possible to the bleeding edge of real-time graphics development.
I remember having long debates with a colleague about what high-dynamic range lighting "meant." "If we spin around in our office chair, our brains do not suddenly change the overall brightness of this office. Why should we be programming games to do this? Why is everyone doing things that way?"
My concerns were of a more fundamental nature as well. What is a diffuse texture? A diffuse texture is a well-understood concept in realtime graphics. Anyone with a basic knowledge of shaders should immediately recognize:
color = lighting * diffuse + ambient
So I turned to research papers. Books. The medical field. Everywhere that was remotely related to possible breakthroughs in photorealistic rendering. Research papers are excellent for assembling techniques, but not results. The books in human vision and color science were more promising, yet most of the industry seemed (and still seems) to pay little attention to them. Compare a book about color perception to, say, http://www.pbrt.org/ and you'll see a stark difference. Flip through the table of contents and you get transformations, shapes, primitives, color and radiometery, sampling, reflection, materials, texture, scattering, light sources, monte carlo integration...
And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image. It's not for lack of processing power. There is a disconnect between the old rules and those that will ultimately result in real-time realism, and you won't find it in that table of contents.
Now, the trouble with writing all of this is that if I knew how to do it, I'd have done it already. It's a life-long search, and it's not so easy to refute an entire industry without being (rightly) dismissed. But if you wish to know what I suspect are the ways forward, it's this: Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature. "Just throw in physical models and presto!" has not thus far been true.
You'll notice, for example, that computer graphics in videogames have plateaued. They get more impressive with each generation, but that impressiveness does not get them progressively closer to looking real. Nor should it. A computer game tells a story. The closer it looks to real life, the more restricted the artists are, along with the rest of the design of the game.
So we turn to the movie industry for hope. But it's restricted in exactly the same way. The research papers are all along the lines of new techniques to try, or studies of existing pipelines and how to deal with their complexities. It's not fundamental research.
As someone who has spent his life in pursuit of realism in computer-generated video, my recommendation is this: Read DaVinci's journal. Pay attention to what each page is saying. He had to discover from first principles what makes a painting look real, and why. You'll notice that he spends most of his time talking about human vision and our perceptions of color.
If someone is going to make this development happen, it's not going to come from the game industry, and it won't come from the movie industry. That leaves you. Hopefully this will encourage some of you to pursue this. Once you accept that most of the computer graphics industry isn't actually focused on achieving realism, you'll start to develop your own techniques. My hope is that this will eventually lead to a breakthrough.
This is actually testable. Some archviz images are indistinguishable from reality for non-graphics experts.
> Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature.
Yes, very important. Art teachers try to reinforce this by saying "seek reference". It's true for graphics as well, and it's possible to be more empirical.
Photos, yes. Not video. It's an important distinction because the path to photorealistic video won't pass through impressive-looking still frames. Visual processing of movement is quite different.
What I thought your original point was is that 3D rendering ONLY cares about photorealism. Physically based renderers have been greatly influenced by photography (both still and film). Think light probes, camera lenses, etc. Much of the post-processing you see in a scene is also derived by what a camera would see, from focusing, to blur, to HDR now, and of course the infamous lens flare!
So I think a physically based renderer which uses the human eye as its camera would be interesting to see more of.
While noble, I cant help but wanting to understand what alternatives you were leading yourself done. As an example, light can arguably simplified to intersecting cylinders and spheres that bounce off surfaces to create new 3d shapes. Each shape also would have an origin 2d shape based upon whats reflecting it. an "eye" reads shape intersections with self and also can filter those intersections in respect to origin shape. After each bounce, the new shape takes form as the bouncing lights color multiplied by the color of the bounced object In low light situations, subtle luminosity differences can be enhanced.
What I did was offer an example. Perhaps youll one day be successful but I got the impression you are some kind of renegade with a mission. While I can certainly relate to that, I view science and building the future quite far from renegade status. And in the mean time, you gave me a sob story with no algorithms/solutions except for "take real pictures and compare them". As a lazy programmer, walking outside and discoveringvthe world doesnt interest me too much.
Here's something specific: What did you mean by "multiply"? You cannot "multiply" colors. Not unless you concede that your model has nothing whatsoever to do with physical reality. And at that point, why not use a photo of nature (or your eyes' perception of nature) as a baseline comparison?
From http://www.feynmanlectures.caltech.edu/I_35.html:
"The phenomenon of colors depends partly on the physical world. We discuss the colors of soap films and so on as being produced by interference. But also, of course, it depends on the eye, or what happens behind the eye, in the brain. Physics characterizes the light that enters the eye, but after that, our sensations are the result of photochemical-neural processes and psychological responses.
There are many interesting phenomena associated with vision which involve a mixture of physical phenomena and physiological processes, and the full appreciation of natural phenomena, as we see them, must go beyond physics in the usual sense. We make no apologies for making these excursions into other fields, because the separation of fields, as we have emphasized, is merely a human convenience, and an unnatural thing. Nature is not interested in our separations, and many of the interesting phenomena bridge the gaps between fields."
Walking outside and discovering how the world looks is exactly how to improve your techniques as a graphics programmer.
Once the info leaves the receptors, now that's a another story...
Your type of examples seems like uts more useful for machine learning. Collect enough samples, build enough models, run enough tests and youll get something accurate-ish.
Light can be multiplied, in a way, through interference.
I actually am a bit confused what you are arguing here.. all of these problems you're mentioning have been well-understood by graphics researchers for the past couple decades.
The reason most of the CG you see in TV or movies looks very good is because they take place within real video. We're not looking at a completely CG scene -- it's mixed with video from the real world. And that's a perfectly valid technique, but my comment was talking about 100% CG.
A secondary point re: the film industry is that artists must necessarily retain control of the art pipeline in order to create scenes that advance the plot. That requires the art pipeline to be flexible. The more flexible your art pipeline, the more productive your studio is. Yet that flexibility is precisely opposite to realism. Obviously, the more realistic a purely CG scene looks, the less flexibility you get, otherwise it wouldn't appear real; hence the argument that the vfx industry won't be the ones to produce the elusive fully-CG fully-realistic video. (It doesn't make financial sense for them to do so, if nothing else.)
Replaces "looks good" with a more generic "gets the right answer" and you'll notice that phrase repeated on any simulation context where computers are not powerful yet to work from first principles.
If you want a safe place to look, look at chemistry. They have simulations that vary in complexity by a huge number of orders of magnitude. You'll see those non-natural techniques that "look good" get progressively applied as the simulation gets slower and slower. It works really well, but once computers are fast enough, everybody just throws them by the window.
1. "Take a picture and make it look right" is exactly what people have done for things like POVRay since at least the 90s. I can't find it now, but at one point someone set up a glass ball on a checkerboard and used a point light and a camera to confirm the diffraction and distortion models were correct because someone claimed they didn't look right or were doing the wrong thing. The math that's there for rays, shapes, diffraction, diffusion, caustics, etc. is accurate, and necessary but not sufficient. Which brings me to
2. CG realism has generally hit the Uncanny Valley by now. It's so close to real that we think "that's pretty good", but it's far enough away that we still know "something's wrong". It's the difference between a dummy, a corpse and a live person.
A couple of examples I remember from the past decade are laser + milk and better skin rendering on one of the NVIDIA demos a while back. There wasn't (isn't?) a good model to simulate the diffraction and subsequent diffusion of a laser shining into a glass of milk. Actual lasers with actual milk don't do the things we expect of modeled lasers in simulated milk. Some component is missing, but all the existing math is right for lots of other cases. The NVIDIA skin thing was adding 3 or 4 layers to an existing model to simulate subsurface scattering and reflection that happens in skin, vs old models that treat skin as paint. The old stuff was right, just not enough.
All of that aside, there are decent photorealistic rendering options for some materials today, but at the cost of CPU hours of render time. If you can do better then please do, even if it's just for one material or one physics action.
To go even further, learn to draw. Use your hands and other senses instead of only thinking. Read Drawing on the Right Side of the Brain by Betty Edwards and The Hand by Frank R. Wilson.
I'm curious what your favorite games are, visually, @sillysaurus3?
A couple of mine are Far Cry 2 and [Elite: Dangerous][1] Horizons. I also love the sound in both of these which is to me a seemingly inseparable element to great screen work.
[1]: http://simongriffee.com/notebook/elite-dangerous-education/
I've been considering an answer to your question for the past couple of days. As cheesy as it sounds, I suppose my favorite isn't necessarily a game, but a video:
https://www.youtube.com/watch?v=ekLEDoitnNg
This was what I grew up with, and what helped inspire me to pursue graphics programming. I used to watch it on loop dozens of times, being amazed at how graphics got better with each game.
That was back in the day when you had to download codecs for DivX, and find these videos on Kazaa... Ah, memories. :)
A couple of mine are Far Cry 2 and [Elite: Dangerous][1] Horizons. I also love the sound in both of these which is to me a seemingly inseparable element to great screen work.
Good choices. Absolutely agreed re: sound. Sound and animation are both vital (but often overlooked) elements of a good story, with all the focus on graphics nowadays.
That really was all about algorithms and sampling.
It's not so much a novelty as it is pushing the envelope.
The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star.
And then add in interstellar dust, which has non-neglible mass. And then run the simulation for 8 billion years.
There are naturally some optimizations and short-cuts to be made. :)
Summary of what's below: The type of simulations described in the article do not scale like O(N^2) but more like O(N log N).
A small correction here.. The gravitational force calculation for N-body simulations are only O(N^2) if they're done using direct summation and force calculation. That's only necessary if you care about the motions of individual stars. The largest simulations I'm aware of that do that use a few 10s or 100s of millions of stars. They're typically used for studying individual star clusters, the regions around supermassive black holes, and perhaps dwarf galaxies. But the simulation in this article/press release has a few 10s of billions of particles, I believe. And those particles are all much more massive than individual stars. So while there are "star particles" in the simulations, it's more accurate to think of them as a set of particles which constitute a monte carlo sampling of the stellar mass distribution.
One aspect of that is you now no longer care about the detailed trajectories of individual particles[0], instead caring about the overall representation of the mass distribution. So you don't need as high of accuracy in the force calculation. Going back to the 1980's, people have been using numerical tricks to reduce the scaling of the simulations with the particle number. This includes algorithms such as treecodes, particle-mesh codes, etc., which reduce the scaling to O(N log N) or even O(N). These are sufficient to run simulations of describing the overall gravitational field.
The scaling for hydro codes is probably more complicated, particularly for simulations which use a moving mesh.
There's a nice summary of numerical techniques for this type of work here: http://arxiv.org/abs/1412.5187
[0] One can determine if interactions between individual stars is important to a system's evolution by comparing the 2-body relaxation time to the age of the system. This is discussed on pages 8–9 of the preprint linked above.
Also, for everyone to enjoy simulations results, check the simulations by Canup from the giant impact hypothesis[2]. Another thing to consider, the supercomputer used for this specific simulation had about the same computer power as a high end GPU from today.
[0] https://benedikt-bitterli.me/bbw.html
[1] http://lasp.colorado.edu/~espoclass/ASTR_5835_2013_Files/201...
> GIZMO is the name of the code suite, not the numerical technique.
Oops, misread that. You already know it's the code suite, not the technique. :)
It's a little bit of both. Over the past 5–8 years, some new numerical techniques ("moving mesh") have been developed (or at least first applied in astronomy) for hydrodynamic simulations[0,1]. These new techniques do a much better job of handling fluid instabilities, which can be important for properly simulating the gas in galaxies (which can have a huge range in densities and temperatures).
Also, people have developed what they call "cosmological zoom" simulations. In these simulations, you run a low-resolution (sometimes dark-matter only) simulation of a fairly large volume of the universe. Then you identify a particular dark matter halo at the end, that you think matches the type of galaxy you're interested in (say, a Milky Way type galaxy). Then you go back to the initial conditions, re-initialize it at very high resolution and include "normal matter" (baryons) and re-simulate the whole thing to study the formation of that particular dark matter halo and its associated galaxy.
Finally, the general increase in computing speed has allowed people to add in more physics to the simulations and run higher-resolution simulations. In this case, it is the addition of a physically-motivated model for energy injection into the gas in galaxies, from the ongoing star formation. This means energy from supernovae explosions and associated shocks, stellar winds, and momentum from scattering/absorption of photons by dust grains[2,3]. This more realistic set of models manages to reproduce the overall efficiency of star formation (what fraction of the gas gets turned into stars over some period of time) _and_ create a realistic temperature-density structure for the gas.
Now that people seem confident that these models can reproduce the overall star formation properties in galaxies, they are starting to see what this type of star formation / feedback model implies for the properties of galaxies. Of course, there's still lots of work to be done in making detailed tests of these models. While they reproduce the overall efficiency of star formation, we aren't yet sure if they can reproduce the shapes and sizes of galaxies. But people are working on that and we should know before too long...
[0] https://arxiv.org/abs/0901.4107
[1] https://arxiv.org/abs/1409.7395