Real-Time Ray Tracing Demo | Hacker News Reader

Real-Time Ray Tracing Demo

unrealengine.com

248 pointsmariuz100 comments

Some context: those real-time renders are done on a $150,000 rig.

Ofcourse when you are a movie production company this will be no problem. But I think most gamers still have to wait a little.

Gamers will still see the effects of this power on lower quality rigs. Global illumination can be approximated by doing ray-tracing computations at the surface level rather than the image level. Modern-day games are approximating more and more complex effects in this way, and I personally think that the gap between a game like Nier Automata or The Witcher 3 (which both use a lot of global illumination but are still rasterized) and ray-traced images isn't that stark.

$50,000, but still quite a ways from consumer level

https://www.tomshardware.com/news/nvidia-v100-dgx-station-gp...

Apparently you can do this with 4x TitanV which is $ 12k total.

citation: https://twitter.com/BrianKaris/status/977224198148931584

This is the case for 5G ultra low latency and edge computing (as is VR).

Move GPU units close to proximity and have 1-2 ms latency and you justify more expensive hardware compared to individual purchases.

During the original presentation they said the demo video was using 2x Volta cards which is $3k each, so $6k in graphics cards. But I'd trust the guy on Twitter more than the marketing pitch.

https://youtu.be/7dqOSTjpfOI?t=3m18s

Interesting, he mentioned it was totally unoptimised at all.

We should be able to get close to 4x the transistor count of Volta @ TSMC 7nm, sometime in 2020.

But the 1080P and 24fps is only enough for movies, Gaming needs much higher then that.

It’s not really enough for movies anymore. 4k is growing in popularity dramatically, and I for one am entirely sick of 24hz.

Ofcourse when you are a movie production company this will be no problem.

Oh, you'd be surprised.

> using the latest GPUs and machine learning techniques

I have no expertise in either graphics or machine learning, but is this just buzzword for its own sake, or is there actually any way that machine learning is applicable to a mathematical problem like rendering?

Again, from my position of relative ignorance, it seems like graphics is largely a well defined problem space that needs mathematical ingenuity and bigger/better hardware thrown at it, rather than building a black box function by training it on sample inputs and outputs (what would that even look like?)

They're using Nvidia's denoiser which is based on a pretrained neural network.

Edit: look up "optix denoiser"

To my knowledge, it's a buzzword. That being said, I'm also not an expert, and would like to hear from one to be sure, since it would be interesting af if they actually are using machine learning for Ray tracing

For raytracing it's been used to de-noise images: https://blogs.nvidia.com/blog/2017/05/10/ai-for-ray-tracing/

TIL, thanks. That's a more prosaic answer than I was expecting but clever and makes sense.

If you look closely, at around 0:22 it causes some artifacts in the rifle's red light matrix thingy. It could be regular compression artifacts, but they look like denoising artifacts to me.

Could it be just an intentional focus effect?

How "local" are such techniques? Is the DL network doing things like "this is a car, and this is a patch of it, so it must look smooth"? Or is it more like "this patch is noisy, so let's make it smooth and hope for the best"?

More on the local end. Traditional denoising is edge finding + blur; you find the edges that you want to keep crisp and you blur on both sides of the edges. ML has been really good at somehow replicating and improving upon that result without explicitly building any sort of understanding about what it is that it's denoising.

I don't think this is what they are actually using, but path tracing can be reduced to a really huge sparse sampling problem. This means that given any predictive model of how much paths through the scene will add light to a pixel, you can make that model unbiased by using https://en.wikipedia.org/wiki/Importance_sampling

So a model trained to take the samples you have so far and predict paths that will also connect eye to light sources could hugely speed up rendering. I don't know of any software that does this, so free idea if someone wants to try it.

there was a paper recently caching sparse radiance results and using ML to predict important paths from that. I believe the latest version of Octane might do something like that if you read between the lines of their marketing guff.

Their current raytracer is limited by the available compute power, so it's forced to use as few samples per pixel as possible, which results in a noisy picture.

However, there's a synergy between their on-chip ML cores and their raytracing engine: they use ML cores (which are typically useless in graphic workloads) to de-noise ray-traced images.

Take a look at the youtube channel 2minutepapers [0], it covers various ML/DL papers and their results. The person creating the videos has a background in writing rendering software. He has various videos covering papers that use ML to improve rendering and graphics. Not long ago he created a own paper about teaching an AI the concept of metallic, translucent materials and more [1].

I really recommend checking his channel, videos are short and fun and easy to understand even without ML background. You can really sense his passion and excitement for this topic.

[0] https://www.youtube.com/user/keeroyz [1] https://www.youtube.com/watch?v=6FzVhIV_t3s

An anecdotal observation: there seem to be many ray tracing-related articles and ShowHNs that make the front page. For someone who isn't in the computer graphics space, what's the importance of improving existing ray tracers or the novelty in writing one's own? Is writing one a particularly challenging thing to do? Is improving existing ray tracers the equivalent of chip manufacturers increasing CPU performance in the chip world? Just trying to understand this a bit better.

The big news is a standardized API to do realtime raytracing. Both of those are a big deal because as you said everyone until now wrote their own and had to overcome the challenges associated with that. If that's taken care of (for the most part), the only things left to write as a developer are how you want to use this feature. In TFA, it's using it to do realtime raytraced reflections where previous techniques to do that in realtime are too much of an approximation and have big drawbacks.

> what's the importance of improving existing ray tracers

In the offline space (film, visualization), cost. Movies get cheaper to make.

In the realtime space (games), lots of visually important effects are difficult to do with rasterization, leading to complicated/large/expensive-to-maintain rendering solutions. With a physically based path tracer, a lot of these effects come out of the box. Path tracing has traditionally been too expensive to do in realtime, however, and has been 'just around the corner' for the past 20 years. If I'm not mistaken, recent advances in image de-noising, which seems to be the crux of the new NVIDIA technology, have made real time path-tracing on the GPU much more practical.

> Is writing one a particularly challenging thing to do?

Depends how deep you dive. You can write a basic tracer to render a sphere and fit it on a business card -- this is a typical assignment in introductory graphics courses. Rendering complicated materials is a never ending rat hole of complexity, lots of rendering research looks effectively like material's science, these days.

>Is writing one a particularly challenging thing to do?

I wrote a simple ray tracer for a college project in the 90s so I would tend to say that part of the novelty is that it's actually quite easy to write a very basic ray tracer and get cool looking images :)

Funny, I bet more people know how to raytrace a sphere in software than know how to rasterise a polygon these days. Neither is terribly hard (or terribly useful, to the average dev.)

To be fair, raytracing a sphere is easier than raytracing triangles.

Even a single triangle is harder than a sphere, but with a triangle mesh you can't avoid thinking about acceleration structures and having a watertight intersection test, which includes worrying about floating point precision issues.

So what you're saying may be less about raytracing vs. rasterization and more about spheres vs. triangles.

Exactly. And, similarly to fractal rendering, it's also the fun of using what is essentially pure mathematics (3-D geometry, algebra, matrices, etc.) to generate pretty images.

I think they're just cool.

First of all, "ray tracing" means at least two completely different things in the world of computer graphics:

MEANING ONE: GRAPHICS HACK

The old meaning of "ray tracing" is a hack: you take your screen, you figure out the projection into the world that represents the "camera", you trace out from each final pixel in the image to see what geometry it hits first, reflect, and allow for a certain amount of bounces before you decide on a color for the pixel based on the surface material properties you interacted with and where you got to a light.

The driving intuition here is that this bouncing this "ray" off geometry has some surface similarity to how light actually works in the real world (Except backwards: we don't shoot photons out of our eyes of course). However, it's not really any more "physically-based" or physics-accurate than traditional rasterization, which is just loads of hacks: i'm a triangle, i'm angled like so towards a light, i should be this color, except i'm bumpy, so lets simulate that by pretending im actually angled a bit off that angle here...

This particular hack does one or two things really well that normal rasterization does not: mirrored surfaces and translucent materials with a high index of refraction. This is why 99% of ray tracer demos include a bunch of mirrored spheres floating over a fountain on a marble floor. Ray tracing, the hack, _does not_ solve many other problems that ignoring the physics of lighting leaves out: caustics (those effects you see on say a the table under a glass of water when it catches the light), soft shadows, ambient occlusion (the tendency for things that are hidden by other things to not receive as much illumination, the way the joint where the wall meets the ceiling is a little darker than the wall.)

Writing this type of ray tracer is actually really simple and easy, which is part of the reason lots of people are interested in ray tracers, I'd wager. It's an intro computer graphics project. It also provides ample opportunity to hack on the performance because it's embarassingly parallelizeable and there are all sorts of intuitive algorithmic enhancements available (octree subdivision, etc.)

A lot of posts get on to HN about this type of ray tracer because the two things these are good at are cool, and given the parallelization opportunities fit with modern hardware development, this is hoving into range of "shiftable from offline to real time" which is cool. The second big reason there's always random ray tracer stuff on HN is because a ray tracer has one of the highest code-volume-to-cool-effect ratios so it shows up in intros and demos all the time. It's a great little project where you get something super neat super fast, but you have loads of runway to make things better with satisfying increments.

MEANING TWO: ACTUAL GRAPHICS SOLUTION

The "real way" to determine the appropriate illumination for every pixel on a rendered image is to start with all the lights, characterize the photons they emit, emit a zillion, trace them around the room bouncing off of objects, and see which ones ultimately hit the camera plane and write those colors. This will produce a photorealistic (or heat-camera realistic) rendering of a scene. You are simulating the actual physics of illumination here.

Of course, this is computationally insane, so there are approaches to dramatically reduce the computation at the expense of accuracy. The family that concerns us here are "path tracing" algorithms (e.g. "Metropolis Light Transport" is a monte carlo approach to integrating the "rendering equation", the equation that determines how things should look under lighting conditions). There are other ways to start with the rendering equation and reduce the computational complexity (e.g. radiosity) but this is the one that is sometimes called "ray-tracing".

These methods handle all the things I mentioned hack ray tracing not handling: soft shadows, ambient occlusion, caustics, etc.

I think this kind of stuff gets to the top of HN because it's awesome? My more serious answer might be that a lot of real-time rendering code is layered with hack upon hack to handle adding bits of detail to a rendered scene to make it more photorealistic. There's nothing particularly physical about bump maps, environment maps, or a zillion other real time shader tricks. They're all cool but they're hacks. The idea of being able to make a path-tracing-type renderer that runs at realtime speeds would mean that we wouldn't have to spend so much time hacking in things like prebaked shadow volumes or what have you in any realtime renderer (e.g. a game engine) and consequently our engines might be much more generic, instead of carefully tuned shader-by-shader to the specific types of effects we want to create. I'm kind of doubtful of this (I don't work in the games or computer graphics industries though so I'm extremely not an expert) because the "look" now is so much a look: people like bloom and lens flare. Film doesn't look "real" but when you make a movie tha looks more accurate people complain it looks "fake": you need to flicker.

There's nothing wrong with starting the paths from the eye; the laws of physics are completely time-symmetric, so you can get equivalent results either way.

The hack is not inverse ray casting. Thing become hacky when the bouncing ray's bounce angle/amplitude etc. Determined by a mapping instead of real geometry/black body etc.

There's even been research into switching a lightbulb + camera setup for a scanning light source + a photodiode. http://graphics.stanford.edu/papers/dual_photography/ (Note there's a video at the bottom of the page that walks through the various techniques.)

It's been a long time since I studied this, and I've forgotten most of it, but I think the key problem is that you don't know in advance what the ray density should be at the eye, so when you start the rays at the eye, you typically emit a fairly even density of rays and see what they do in the scene, but this won't give you correct results for some scenes, where paths emitted from light sources should end up more concentrated in some parts of the eye/image than in others.

There are other hacks too - the typical start-at-the-eye technique doesn't bounce the paths around the scene until they hit a light, it just does a couple of bounces, and at each surface it takes an estimate of how lit it is by looking at the lights that are visible from the intersection. I guess you could simulate the rays bouncing until they hit a light source, but at that point, why not just do the simulation from the light source like in the physical world.

What you really want to do at a surface intersection when you're going from the eye is an integration over all the possible paths light could take to hit that point and reflect along the path you took from the eye. For specular highlights and perfect reflections, this is reasonable, but in other cases there are an infeasibly large number of possible incident paths that could come out along the path you took.

I've always heard of this distinction being "path tracing" as your #2 and "ray tracing" as your #1.

> Is writing one a particularly challenging thing to do?

Depending on the speed at which you want frames to appear (vs. noise), it can be. Writing a raytracer that can render a few shiny spheres in a few seconds on modern hardware is trivial - not more than a weekend project if you know nothing about the subject. Writing one that can render a Stars Wars scene 25 times per second is a vastly different beast.

Simply put, raytracing is one of the most computationally expensive operations in real time graphics (games). Improving raytracing allows moving game graphics more from prebaked artistry towards physical simulations that still result in beautiful visuals.

You can write a ray tracer in a weekend.

http://in1weekend.blogspot.com/2016/01/ray-tracing-in-one-we...

Improvements to commercial render engines today are often these days about adding new features and improving end user workflows. Here's last year's SIGGRAPH talk from Renderman. https://youtu.be/iDUV4ISCklA

Commercial end users include digital art studios and other advanced users. In-house teams of specialists will use low-level APIs to achieve desired results. Render engines often more like a software development platform than a video game engine. The CG industry is constantly changing, these days a lot of effort seems to be aiming toward standardization and interchangeably as pipelines become more complex and software becomes increasingly specialized. (Zbrush, Maya, Mari, Houdini, Katana, Nuke.) A render engine will have to support several of these DCC packages and ensure artists get consistent results.

Render engine speed improvements still play an important role in development, but it's most noticeable in areas that are outside of the traditional "path tracing" parts. This is things like subsurface scattering (the red glow of a flashlight behind your fingers), volumetrics (smoke and steam), and caustics (shimmering light on the table when a water bottle sits in the sun). The "ray tracer in a weekend" parts are very thoroughly optimized, but you do see innovation now and again with developments in importance sampling methods. What you say is true in this sense: modern render engines are still just as CPU intensive as ever and you still need massive server farms to create final frames.

A more modern development is GPU-based render engines such as Redshift, Octane and Blender's Cycles. These engines are subject to the limitations of GPU memory - unlike a server where you could pop in a TB of RAM if necessary, GPU render engines are not currently able to handle extremely complex scenes like hero assets with tens of GB of textures, entire cities worth of geometry and so on.

This Unreal Engine feature is a slightly different world. This isn't a raytracer in the sense of what I've been describing above, what it's doing is adding a layer of reflective effects, computed with raytracing, on top of ordinary "game engine", where in theory you could add controls to manipulate the camera with your mouse and keyboard and look around.

To hit 30fps, the trick is adding just a little bit of reflection to improve the look, and setting up assets do that the approximated effect looks appealing and convincing. The machine they ran this on would cost nearly $100k, though the cost of creating professional assets specifically tuned for this specialized environment would rival that. However for an architecture studio trying to make a pitch showing a stunning interactive demo of their design, the economics are nearly there. And on a wider scale, this same technology could add extra pop to glass, metals, and liquids in next gen video games on a more modest gaming rig.

The quality of ray traced scenes is much better and much closer to the real world than rasterized scenes. Yet ray tracing is very expensive in terms of computation (often too expensive for real time ray tracing).

So we have some technology which delivers superb results but is some sort of 'future technology' as we can't afford it yet. I think that is whats makes so many interested in ray tracing here. We know that some day that technology will revolutionize the rendering industry and every time some progress is presented we are curious how that progress is being made.

To fully understand the obsession with Ray Tracing, simply download Blender and run the Gooseberry Benchmark. Gooseberry is Blender's hardest benchmark. It represents a typical scene in the upcoming Gooseberry open-source movie project. So its a "real" scene, representative of what movie studios have to deal with.

On my 16-core / 32-thread 1950x Threadripper, it takes about 35 minutes to complete the benchmark.

That's 35-minutes to calculate ONE FRAME of the movie. It'd take about 300-days to have my computer calculate the entirety of the 10-minute long movie.

There are huge clusters of computers out there in Disney or any other animation studio, whose sole purpose is to calculate the light paths and properly shade these movies. After all, computers are cheaper than animators.

------------

Performing these calculations faster is a MAJOR hurdle in the movie industry. Gamers hope to eventually get these graphics into video games, but the math is just too computationally expensive to run right now.

---------------

Overall, the algorithms behind raytracing are relatively simple. If light is pointing at an object, it should be brighter. If light isn't pointing at an object, it should be darker.

There's a bit of details in the matter. Specular shading vs Diffuse Shading vs Subsurface Scattering vs Ambient Occlusion. But that's the artist's job. The artist simply tells the computer which algorithm (or combination of algorithms) best emulates the material.

The computer programmer's job is to simply emulate the light as it bounces off of the objects. That's all it is, very simple algorithm (ray-tracing) applied to a lot of points in 3d space, in a way that has been documented for decades.

The math has all been documented. It just requires a supercomputer to handle all of these calculations. So its a task that requires highly-optimized coding and really, really big computers (or clusters of $10,000 GPGPUs). You know, the kinds of stuff that gets ycombinator readers really excited.

-----------

Diffuse Shading -- when light hits a "rough" object and it doesn't bounce very cleanly. This provides the majority of the light in most cases. When held directly against light, the object is brighter.

Specular Shading -- when light bounces off cleanly the surface of the object. Leather and Metals are often a high degree of Specular.

Subsurface Scattering -- when light is absorbed by the object and bounces out elsewhere. This is very common in human skin: red light is often absorbed by the skin, and the light path will be diverted elsewhere.

Mirror -- Mirrors are incredibly handled well in raytracing, and are handled poorly by standard techniques. Expect lots of reflections as devs unleash mirrors upon us! (Kinda like how Wizard of Oz had overly bright color design because film-makers were flipping out about color movies).

Lots of other algorithms -- Smoke, Clouds, Fog, etc. etc. Its the Artist's job to know all of these algorithms. The Programmer / Raytracer just does whatever the artist requests.

Water is a great example that uses a combination of these. Imagine an ocean for instance.

Water is partially reflective (Mirror). Subsurface Scattering of the blue-wavelength occurs, so you let blue-light enter deeper into the water, but reflect off red/green. The top-layer of water is incredibly shiny (specular). Finally, ocean waves creates "foam", which is well represented by diffuse shaders and textures.

> It represents a typical scene in the upcoming Gooseberry open-source movie project

Gooseberry has been released -- https://www.youtube.com/watch?v=Y-rmzh0PI3c ... they have done multiple movies since then including agent 327 https://www.youtube.com/watch?v=mN0zPOpADL4 and Hero -- https://www.youtube.com/watch?v=pKmSdY56VtY

GPU acceleration really helps cycles (Blender's open source raytracing engine)... however, the gooseberry benchmark uses up too much ram to make this render purely off the GPU.

Writing one is fun because it’s trivial to start, but you can be sure that there are features that are simply beyond your capability almost no matter how good you are. And it gets hard fast. It’s also very cross disciplinary if you like math, physics and system level programming. Those of use who wrote ray tracers are probably the same that code toy synthesizers. We like numbers. Once you made one it’s also an excellent way of learning a new language to make a new one.

Nice challenges include:

add parallellism. Traversing a tree hierarchy efficiently with SIMD or GPU code isn’t trivial. (I tend to dodge this part of the problem by using intels simd lib Embree)

Or make it spectral (trace with wavelengths instead of simple colored light, to get effects such as rainbows in refractions). This is actually reasonably simple - but then you realize it ruins your hope of parallelizing efficiently as the rays diverge when they refract!

Or implement realistic subsurface scattering. Doing it physically right means your render will never finish, so it’s back to inventing the most clever hacks - just like rasterizing!

> your render will never finish with physically correct subsurface scattering

did you mean "will take a very long time", or "won't actually terminate"? if the latter, why? that sounds really interesting!

Path tracing in general do not technically terminate since it's trying to approach an integral (the rendering equation) with sums. It's up to the user to determine the sample per pixel threshold with which he is satisfied. The goal is often to not have too much noise for the time spent. It's not easy because noise reduction over time tends to be logarithmic.

This. Doing “physical” SSS means shooting rays stochastically into the medium (skin, milk, marble) and having them bounce around inside, emerging at a random location, but more likely close to where it entered.

That type of tracing will simply not produce a reasonable result in a reasonable time. You have to cheat.

That's not completely true - at least in the VFX industry we've been doing brute-force SSS (with multiple scattering) for more than two years now in some cases. Yeah, it can be slow depending on the MFP, but it's usable...

Cheats like Dipole/Dwivedi are still used when they can be, but they often have artefacts and aren't completely correct, so they're starting to be avoided now.

Interesting, any particular renderers that do this?

Dwivedi is brute force. It’s just a sampling strategy.

True, but it's biased, which still causes trouble with some edge cases, depending on how/where you bias towards - biasing towards point-of-entry isn't always better than not biasing depending on where the lights are, and it's cumbersome to set up for artists (depending on where you bias towards).

To add some challenges to your list:

- implement a bidirectional path tracer to increase convergence speed

- implement multiple importance sampling. It's a simple concept that I found astonishingly hard to implement in practice.

- implement an acceleration structure (like a BVH) that builds and caches efficiently. Can you make it fast enough to have animation?

- implement shader based unit tests to ensure correctness. Thus is also surprisingly hard.

Depends which market you are in.

First for a ray tracer to be effective it must model billions of photons, so optimising how and where you fire them is key to making a _fast_ ray tracer. A non optimised, (therefore multi hour per single image render time) ray tracer is fairly easy to write (can be done in less than 5k lines of code)

As for what the point is, there are a number of things: Raytracing means that you are effectively simulating the physical properties of something. This means that light interactions between objects come for free (well, kinda) When you put a lamp shade on a light bulb, it will cast shadows. Windows are transparent, etc, etc. In raster land, thats not a given.

This means that architects who are designing building in cad, and have been very specific with their material choices can get accurate representations of how the light will interact inside the building. Maxwell render bills it's self as a physically accurate renderer, but it is _slow_, very pretty, but slow. (yes, there are trade-offs, a real-time renderer is not going to be physically accurate, but it is way more accurate than a rasterer) So making small incremental changes might mean waiting for a number of hours for the render to finish.

In VFX, renders can take > 24 hours per layer, per frame (Each finished frame might have > 1000 assets, each with > 100 versions, each with a number of layers.) So any time saving there can save thousands.

Also, getting real time feedback makes production much more fluid and intuitive. At the moment, when acting with CGI characters or backgrounds, you have to use a standin: https://imgur.com/gallery/OVk4DYp

This means that the cameras have to be much more scripted, a lot of working out has to be done to make sure that CGI elements blend in properly. In the film gravity, they used realtime motion capture heavily, as most of the elements (in some case only the eyes of the actors are real) are CGI. The director is a visual person and wanted to look through a camera and see the space station. He would have sacrificed his left testicle to get the demo's level of realism in his view piece

In comparison to rasterization, ray tracing does a perfectly good job of doing correct reflections.

However from a psychovisual standpoint correct reflections are the least important thing to look for. Just use a dedicated cube map for a scene and use environment mapping and focus on stuff that really matters and everybody immediately notices. Like high resolution textures, some proper anisotropic filtering and "HDR" lighting.

Ray traced shadows are noticeably better than rasterized shadows.

In practice, ray-tracing will never fully take over rasterization. At best, a standardized ray-tracing API will perform a few calculations which are superior in the ray-tracing style.

IE: Mirrors, Ambient Occlusion, and Shadow generation. Everything else (texturing, among other things) will likely remain rasterization.

Why wouldn't some form of ray tracing or path tracing eventually take over once we have the necessary hardware and software and standard APIs to do it at high resolution and high framerates? What features does polygon rasterization give us that can't be replicated any other way?

If there is a limit to FPS and resolution where we say “ok that’s enough” then we could ray trace more with the yearly advances in computing power. And I really thought at one point that was 1080p. But later I thought it was 2k, then 4K, then 2x8K (per eye in VR).

Now I honestly have no idea what the “endgame” is, but if it’s say 8K@120 for single screens and twice that for VR then thats two to there orders orders of magnitude away from what a regular gaming card does today. The problem with numbers that big is that it’s hard to say what the tech landscape is.

The killer feature of rasterization is that it gives an illusion of correctness at a millionth of the computational effort. Apart from efficiency I can’t see many attractive properties.

For VR at 120fps you can do eye tracking and only render part of the image at 8k and keep the rest at much lower resolutions. Interlacing 8k images at 120 FPS is also viable.

Also, resolution has increased really slowly over time, people generally don't really care that much. 1080p is 20 years old at this point.

Pro games are often played at 2K@240 now; it never ends

If you can do ray tracing fast, you can do rasterization faster. And so if the ray tracing is not critical, you're wasting time computing it.

I don't think it's a waste of time. Ray tracing puts you on the path to real-time global illumination, which I think everyone is going to want if it's in reach. It also makes a lot of non-graphics tasks easier, like determining line-of-sight visibility between arbitrary points in the scene or doing collision detection.

Besides, computational power isn't something we necessarily need to conserve for its own sake. We stopped using Gourad shading when the hardware became fast enough that we didn't need it anymore. Polygon rasterization will stick around for a long time I think, but I don't think it will always be an essential part of everyone's real-time rendering technology stack.

http://www.paulgraham.com/hundred.html

> I can already tell you what's going to happen to all those extra cycles that faster hardware is going to give us in the next hundred years. They're nearly all going to be wasted.

> I learned to program when computer power was scarce. I can remember taking all the spaces out of my Basic programs so they would fit into the memory of a 4K TRS-80. The thought of all this stupendously inefficient software burning up cycles doing the same thing over and over seems kind of gross to me. But I think my intuitions here are wrong. I'm like someone who grew up poor, and can't bear to spend money even for something important, like going to the doctor.

"Real time rendering" mostly means games, and performance is a very very easy bottleneck to hit for games. Because it is easy to make things more fun and interesting by scaling them up: more entities, more interactions, more options, etc.

So I believe you are quite wrong. It is trivial to scale-up a simulation to the point where you're at a performance bottleneck. The only reason we don't scale things up today is because our computers are far too slow to handle it.

And we stopped using Gouraud shading because it is horrendously ugly. We'd still be using it still if it looked half as good as rasterization does.

Gouraud shading and rasterization are not mutually exclusive. Rasterization is a technique for getting from polygons to pixels, gouraud shading is a technique for assigning colours to those pixels. Even flat shading is usually done with rasterization.

One of the surprising characteristics of ray tracers is that if you've got a reasonably efficient acceleration structure, they're relatively insensitive to scene complexity. Say you're rendering ten thousand triangles at a certain framerate, and then you bump the resolution of your meshes up so that you've got a hundred thousand triangles. The framerate drops a little bit, like maybe 20 percent or so, but it's nowhere near a linear slowdown.

You can always slow the rendering times way down by adding a lot of reflection and refraction, but those are things you can't really do in a general and physically-correct way in a polygon rasterizer.

Not necessarily - rasterisation can suffer from "overdraw", which raytracing doesn't (although it has its own issues with acceleration structure requirements, especially where moving objects in the scene is concerned, so it's not a complete win).

One of the reasons VFX has moved to raytracing/pathtracing is that it copes with complexity a lot better in general (we're talking scenes with 500 million uninstanced triangles).

It also allows you to do much more accurate camera projections like spherical / fisheye (which could be useful for VR). Doing things like omni-directional spherical 3D projection in rasterisation requires tricks or warping in 2D, whereas it's pretty trivial to trace rays from the camera and do it completely accurately with raytracing.

Achieving "photorealistic" results via rasterization requires a lot of dirty tricks and slight of hand. Supporting it requires a lot of (expensive) hard work from graphics programmers and content artists.

The content pipeline is relatively simple for ray-traced content: just define the material properties of the surfaces of your scene, and the transport properties of the media your light is flowing through, and the light bounces where it bounces.

The limiting factor for real-time graphics applications is generally the cost of content creation. Real-time ray tracing could mitigate that cost.

Ray tracing WILL DEFINITELY fully take over rasterization once hardware capable of doing it is commoditized.

If for no other reason than the huge amount of effort it will save game developers and game engine developers. Not needing to perform all those hacks to approximate what offline renderers can do. Not needing to write those features into engines at all... It will be so much simpler.

> Ray tracing WILL DEFINITELY fully take over rasterization once hardware capable of doing it is commoditized.

That's not what this demo is about, and what you wish for is decades away at best. DirectX realtime raytracing relies upon a rasterization step to estimate the light for most of the objects on the screen.

https://blogs.msdn.microsoft.com/directx/2018/03/19/announci...

--------------

> Not needing to perform all those hacks to approximate what offline renderers can do.

The Gooseberry benchmark (Blender's hardest test) takes 35 minutes to render on my 16-core / 32-thread Threadripper 1950x, while GPUs simply crash because literally one frame of animation requires 12GB of RAM (far more than whats available on commodity GPUs).

Offline renderers are about 5-orders of magnitude too slow for realtime and about 1-order of magnitude of RAM usage too large for conventional ~2GB or 4GB GPUs.

We're at the stage where SOME simplified raytracing effects can replace SOME aspects of the rasterization pipeline, and only if you buy 4x Tesla V100 (Ohhh... that's more expensive. $8k+ per Tesla V100 == $32,000 on GPUs alone). Furthermore, there were only ~1 to ~2 rays per pixel. All the ray-tracing effects were processed through a neural-network to denoise, and they also average those rays through history (so this technique only works on slow-moving objects).

That's what this demo represents. A LOT of compromises, but hey, they achieved a very low-end realtime raytracer and integrated it into a standard rasterization pipeline to provide mirroring and ambient occlusion.

Its an impressive feat for sure. But we're no where near reproducing movie-quality / offline quality in realtime (~64 or ~1000 rays per pixel).

I didn't say we we're close. I said raytracing will eventually replace rasterization. And it will.

Textures, antialiasing, and applying filters to make the rendered image look nice are all good, but I would argue that what really matters and everybody immediately notices is the realistic modeling of ambient light. Proper global illumination is often what makes the difference between a sort-of-realistic looking scene from something that's hard to distinguish from a real image. The best known methods of doing global illumination are all based on tracing a lot of rays.

Physically correct reflection and refractions are nice to have, but those aren't the only reasons to prefer ray tracing.

All those things are great - but nothing kills my immersion more than very very reflective things (cars, water, shiny metal) not actually being reflective.

Raytracing would bring that.

"ust use a dedicated cube map for a scene"

Most scenes with baked lighting these days many (sometimes hundreds or thousands of) cube maps. One per scene tends to look really bad.

Unreal Engine (without the raytracing stuff) uses something similar to this:

https://seblagarde.wordpress.com/2012/09/29/image-based-ligh...

Cubemaps are blended together and traced against simple box and sphere geometry.

> However, for live-action film and architectural visualization projects where photorealism is valued over performance, ray tracing gives an extra level of fidelity that is difficult, if not impossible, to achieve with rasterization.

Don't essentially all modern production renderers use path tracing? Mental Ray was the last 'major' renderer I know of that used ray tracing, and it was discontinued a few years back.

What is the difference between path tracing and ray tracing?

Roughly speaking, ray tracing shoots rays from the camera frustum, bounces them off geometry, and sees what lights they hit. Path tracing is Monte Carlo integration of the rendering equation. [0]

It is easier to get soft shadows and global-illumantion-like effects with a path tracer. I remember reading articles circa 2013/2014 about Pixar moving to a path-tracer, which significantly reduced the amount of time they had to spend manually tuning lights and hacking in shadow effects, etc.

[0] https://www.quora.com/Whats-the-difference-between-ray-traci...

It's important to note that Pixar wasn't moving away from a ray tracer, but from a rasterization-based renderer.

These terms tend to be used interchangeably. In the context of your original quote I think path tracing is the more accurate term. Even Arnold Renderer is described as a raytracer. ( https://en.wikipedia.org/wiki/Arnold_(software) )

Poking around there does seem to be a fair amount of overloading of the term! Even Arnold's website jumps between using ray tracer [0] and path tracer [1].

[0] https://www.solidangle.com/arnold/ [1] https://www.solidangle.com/about/

I would assume any good path tracer is a good ray tracer as well. If you start with a simple deterministic ray trace you can probably layer on a path tracer that doesn't require as many samples to produce a good image that's fast.

I like to see "path tracing" as a higher level rendering algorithm built on top of ray tracing, and "ray tracing" as a visibility query in a scene using a straight line. Ray tracing is more of a computational primitive that can be used for direct lighting, or for reflections, or for global illumination, or if you want, even for simulating audio.

Note that "path tracing" is also overloaded, some people use it to mean any ray tracing that does global illumination, but the term also refers to a stricter definition which is an algorithm that integrates lighting in n-dimensional path space. A path tracer under that latter definition doesn't do any branching along a path, instead it builds complete paths from camera to light to calculate the contribution to a pixel. This is where the name came from.

The main differentiator here is that this is able to deliver real-time performance at 24 fps -- not much interesting to current PC gaming standards (I'd trade detail quality for 144fps any day), but I guess that this has a huge potential for next gen console gaming.

Why didn't the gaming industry go with path tracing anyway? How much more performance does path tracing need compared to ray tracing, if all else is equal?

> The Reflections GDC demo ran on a DGX Station equipped with four Volta GPUs using NVIDIA RTX technology. For now, you will need similar horsepower to leverage real-time ray tracing. In the future, we can rely on Moore’s law to bring down the cost and complexity of how this technology is packaged and delivered.

While GPUs are getting the full benefits of Dennard Scaling, much more so than Intel CPUs a la Goordon Moore, everyone is feeling the process pain of 10nm, 7nm, and beyond.

It stands to reason that the major advances in real time ray tracing will be evenly split between more efficient designs and smaller transistor sizes.

It seems foolish in 2018 to "rely on Moore's law" to bring a 4 x Volta GPU requirement to something more palatable for the mass market.

Who says that we can't just drop the cost of maxing large 14nm Finfet dies and get some sort of stacking, possibly with low-tech, i.e. single-patterning, silicon interposers? The cooling/power density is not a fundamental problem, and as long as we stay under about 20-50W/cm^2 == 200-500mW/mm^2. At that heat flux we'd need 1-5mm spacing between the hot side and the other side (I do not know about experiments that cover two closely spaced hot surfaces, as you would have in slightly-spaced vertically stacked dies), and would just pump coolant with a boiling point 20K under the die's surface temperature through this. On the other side we'd get a mixture of vapor and liquid coolant, and the vapor could, at least at temperatures that a human can endure, be condensed with a small radiator and a fan, or even a larger, fanless radiator.

The primary reason for such exotic packaging/stacking would be that chips need way less power to transmit over such short distances, compared to longer reach systems, as for a comparison between DDR4 and HMC [0], HMC uses about a third the power for DDR4, and that at much higher clocks. Also compare the AMD Zen IFOP intra-package links, which uses 2pJ/bit for the transceiver [1]. This should be possible to extend to an edge-mounted package (where the processor dies have contacts on one or two edges, and are soldered at a right angle to a suitable interconnection/mounting plate), though apparently no one really tried it. The downside would be that one does not have as many pins on the processor dies as currently normal, but I doubt that would be a significant problem, considering the integration benefits. A possible alternative in case of significantly improved yield would be wafer-scale integration, using the space outside of the normal reticle limit for routing larger feature-size mesh interconnects and handling the necessary routing around this later, but that would result in non-trivial software, as it would have to adapt to a given processor's routing layout. TSMC is apparently able to manufacture with reasonable yield individual 'cells' of slightly over 800mm^2, on 300mm wavers. This comes out at about 90 dies, of which some will be dead. With the power density of 50W/cm^2, that would be 3600W, but the upper limit, with significant efforts (ways to increase the flow velocity on the surface) to increase the maximum cooling capacity, would be around 100-300W/cm^2, which would put this at the TDP of an average 42U rack. The weird part would probably be some kind of reverse flip-chip packaging, where contact modules would be bonded to the functioning dies on the wafer, which would then later be fitted with some sort of cable or so, or they could be a miniature optical module. This would also need to provide the power. Cooling would be done on the back side, but the front would likely need some structural support to hold the static/dynamic pressure of the coolant without flexing/breaking.

[0]: https://www.semiwiki.com/forum/attachments/content/attachmen... from https://www.semiwiki.com/forum/content/2731-did-you-miss-cad... [1]: https://en.wikichip.org/wiki/amd/infinity_fabric#IFOP

> Now, leveraging the efforts of several technology companies, support for Microsoft's DXR API and ray-traced area light shadows will be available in the main branch of Unreal Engine 4 by the end of 2018.

MS only? No, thanks.

How many NVidia-only features do you use? Nvidia today are just as evil as MS was 20 years ago, they just don't have the market share Microsoft did, so no one notices how unbearably evil and unnecessarily proprietary and exclusionary they are. Just like MS was.

I don't, I have AMD :) And I agree, Nvidia lock-in is as bad as others.

Epic is also providing the software rendering engine for the new Millennium Falcon ride at the Star Wars expansion to Disney. This real time ray tracing demo was probably part of the development process for the ride.

https://wdwnt.com/2018/03/first-on-ride-image-revealed-of-mi...

How do you do diffuse lighting with ray tracing? Wouldn't you need an unreasonable number of rays to model it accurately?

You do, the solution is to cheat. Use the same phong lighting model as you do for traditional rendering and add a reflection component via ray tracing.

(Or at least that's what I did when I wrote a real time raytacer)

For non-real time applications the solution is to use an unreasonable number of rays.

Depends. The diffuse direct lighting from a point light only requires one ray, same as with rasterization. Diffuse bounce lighting may need more rays. Area lights may need more rays.

Since both of those things are lower frequency signals, they often don’t need to be re-computed independently for every pixel, so people have various algorithms and caching schemes that drastically reduce the number of rays needed. All of these names do somewhat similar kinds of caching of bounce lighting: photon mapping, irradiance caching, light maps, light baking. Additionally, there are tricks to get decent estimates of shadows from area lights in fewer samples. The demo-scene ray marching guys usually do a great job with one ray (not ray traced, but compatible with ray tracing and very fast.)

Sometimes though, people do use an unreasonable number of rays to render global illumination, area lights & diffuse lighting.

I lost it at the starwars elevator music.

I always chuckled at how imperial armor is so plasticy, and they only doubled down on it since the trilogy.

This ray tracing demo just about caricaturizes it.

Some modern armor is made out of UHMWPE plastic.

https://en.wikipedia.org/wiki/Ultra-high-molecular-weight_po...

Armor tested:

https://www.youtube.com/watch?v=UsjOV7tcGRI

NVidia, please don't tease us, release Volta-based gaming GPUs already!

There are a lot of inaccuracies in the article.

> "Rasterization starts with a particular pixel and asks, 'What color should this pixel be?' "

Nope. Traditional polygon rasterization just works its way through a drawlist and rasterizes each polygon into a frame buffer (checking against a depth buffer so that things far away don't get drawn in front of things that are nearer). It's only at the end of the drawing process that we can inquire what color a particular pixel is.

> "Ray tracing works from the viewing angle and light sources and asks, 'What is the light doing?'"

No, ray tracing traces rays out from the eye and figures out what they hit. Path tracing traces rays out from the eye and samples a bunch of random rays when they hit something, to simulate global illumination, soft shadows, and similar effects. Photon mapping literally simulates photons by tracing rays randomly out from the light sources and depositing their hit locations in a data structure. (That's probably the most literal match for the description of figuring out "what the light is doing".)

> "Ray tracing works by tracing the path of a light ray as it bounces around a scene. Each time a ray bounces, it mimics real-life light by depositing color from earlier objects it has struck, and also loses intensity. This depositing of color makes the sharp reflections and subtle, realistic color variations that, for certain types of materials and effects, can only be achieved with ray tracing."

This is a pretty accurate description of photon mapping. Ray tracing and path tracing work in reverse, by tracing rays from the eye. Since path tracing and photon mapping fundamentally work by tracing rays (i.e. doing ray-intersection tests between rays and objects), it's not completely unreasonable to refer to a path tracer or photon mapping renderer as a ray tracer, but not all ray tracers work the way the article describes.

> "Because it mimics light’s real behavior, ray tracing also excels at producing area shadows and ambient occlusion. "

You can do area shadows and ambient occlusion in a ray tracer if you're willing to do a lot of random ray sampling, but it's not inherently a characteristic of ray tracing.

> "Until now, ray tracing was implemented only in offline rendering due to its intense computational demands. Scenes computed with ray tracing can take many minutes to hours to compute a single frame, twenty-four of which are needed to fill a single second of a film animation."

This may be an accurate description of path tracing with enough samples to drive the noise down to a level that's imperceptible, but ordinary ray tracing of non-pathological scenes doesn't usually take that long these days on modern hardware.

People have been doing real-time interactive ray tracing for a long time (I recall running Outbound on a Core2 duo laptop about 10 years ago. Framerate was choppy and resolution wasn't great, but it worked). Brigade is a more recent real-time path-tracing rendering engine, and it's been around for years as well.

That isn't to say that their demo isn't an impressive technical achievement.

Once we have sufficiently good HMDs (maybe Oculus 5?), we could hook on up to a supercomputer running a version of Luxrender, as that should be capable of providing the necessary physical realism to get over the uncanny valley. Of course this is not feasible for constant usage, but more of a technology demo. I believe we have the necessary technology to get this working within a year, provided the Luxrender guys get some engineers with experience in clusterizing existing software. This could be the visual part of a holodeck.

HMD's are a pretty good solution to VR, but there are still a number of unsolved problems which can't be addressed by higher pixel density, faster refresh, and more powerful GPUs.

For instance, how do you account for the focal adjustments your eye makes when you look at objects at different distances in a virtual scene when you are staring at a screen only a short distance from your face?

Will they support Vulkan/Khronos' alternative, too?

Is there any? I thought RTX is a technology demo by NVidia right now as is DXR...

The PowerVR guys have had some path tracing extensions to OpenGL and Vulkan that look really similar to Nvidia's solution.

Nvidia: http://on-demand.gputechconf.com/gtc/2018/presentation/s8521...

AMD: https://gpuopen.com/announcing-real-time-ray-tracing/

https://gpuopen.com/gdc-2018-presentation-real-time-ray-trac...

Reply on news.ycombinator.com