Neural Supersampling for Real-Time Rendering

This is inaccurate.

On Nvidia cards the RT cores are the ones handling BVH traversal and ray-triangle intersection operations.

The part of the hardware that is running the DLSS ML model are the tensor cores. But the algorithm and the model itself is not baked in, it is provided in the driver and/or game

This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR.

ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable.

Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms.

Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver.

This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.

Right. If it's 18ms before you get any result you can't do anything & that's the lower bound on the FPS you can display. The problem is with respect to the pipeline granularity you're thinking about. I'm not suggesting creating a deeper pipeline. I'm saying make the granularity smaller. Can the app simulate the portion of the world that's actually needed for display? Can the rendering engine get you the result for the first 1/4 of a frame in 1/4 of the time it would take to do the whole screen? Can the display pipeline understand portions of the same frame being submitted at different times? Ultimately can you drive different parts of the display at different refresh rates based on when those pieces arrive.

I'm not trying to make this seem like a trivial thing. The single global view is easy to reason about. We need fundamental research into this space on how to build physics & rendering engines that can unlock this kind of granularity. There's massive data dependencies that are really challenging to untangle. Each layer in the pipe, from physics engines & rendering engines, to display pipeline APIs, to GPUs likes to treat everything else like an opaque system with each providing a complete result to the next. There's an enormous amount of existing architecture that would have to be rewritten. I'm also 100% clear that it's entirely possible that parts of the pipeline can't be segmented in this way to get good looking result or it might make the SW burden on content creators to costly. I still believe it's the right direction though & is the only way you're going to ever see 8k displays @ 120hz or above. Well, that's not strictly true. The other trick will be to use motion estimation & neural nets to provide the interpolation when you drive it higher than the content can support - you can see that with all the motion estimation & supersampling improvements that various vendors are producing.

I don't mean to say that a single frame can take 100ms to render & that you can somehow add more pipeline stages. That's silly because the slowest single stage of your pipeline is the bottleneck.

I was instead saying that if you could re-architecture the entire system end-to-end so that you could generate 1/10th of the frame in 1/10th the time, & there wasn't a resource contention between frame (i.e. you would start processing the next frame as soon as you're finished with the current one), then even though it took 100ms for the entire frame to scan out, each individual section is still updating at 100 fps. Similarly, if you could get update the display in pieces rather than the one-shot serial operation as it is now, then you'd see some serious changes to our understanding of latency & pipelining.

This is all easier said than done & was a total strawman taken to an absolute extreme. However, if the SW could get that pipelined, HW vendors would have a much easier time making their end of it more efficient.