Raytracing won't simplify AAA real-time rendering
c0de517e.blogspot.com
c0de517e.blogspot.com
If your game is close to a starting template, it makes it fast to create something fun and let you focus on iterating with players. The further you are, the more effort will be involved. At some point, the effort becomes equal or greater to the other platforms, however, most kids learning on Roblox don't have the skills to start with Unity or Unreal Engine.
Taking a step back and shifting markets back to GPU hardware, NVIDIA CUDA is Roblox of the GPGPU world - a single stop shop for really great templates and tools to get you 90% of the way to your scientific goal. That last 10% can actually be more like 90% if you're in an area where the platform is missing something (this is universally true for all platforms).
The computing industry is full of tradeoffs and people re-learning and re-creating patterns to solve similar problems to those that were solved 5, 10, 15, and 20 years prior.
If that was true there wouldn't be so much logic and code bugs in AAA titles.
From what I've seen game industry has terrible software engineering practices - why have automated testing when your model is crunch to release and then leave a skeleton crew fixing the bugs after you shipped.
Also being stuck in C++ doesn't help either, an ecosystem with bizarrely the most complicated frameworks I've ever seen (eg. boost) and yet the worst tooling out of anything I've used (with comparable adoption rate).
Those are pretty different concerns tackled by different groups of people on AAA projects. The art requirements can be an order of magnitude larger than the gameplay side of things. That doesn’t make the gameplay side of things easy.
I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices.
With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.
Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others.
>With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.
There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far.
Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)
It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.
So I came in, throw it all away and built within 2 days a deterministic, fully tested and testable stone engine that would implement all the effects management wanted to see, some of which would have been close to impossible to implement with Unity's physics engine. The idea is basically to use the smallest time unit you want to support, and then base all algorithms on that. You don't work on milli-seconds, because that is for the engine to fill out. You work on seconds, for CandyCrush at least. Each time step is basically a second. The time in-between is filled out by potentially non-deteministic animations and particle effects and whatnot. But every second, the whole scenery will synchronize with the deterministic engine that drives it all.
While I didn't have the chance to try this on shooters or MMORPGs, I think especially for MMORPGs, this would be a perfect fit. It solves sooo many problems, I would probably need a day listing all the benefits. Are there drawbacks? Probably, I can't think of one right now, except that it goes against anything normal game developers believe in.
I think at some point, the gaming industry forked away from solid software engineering practises.
I'm a former game engine lead from an Activision studio. The whole engine architecture was my responsibility, testing included.
You can easily write tests for all the deterministic stuff; math, physics, save games, utility code, filesystem code, etc.
However, where it gets difficult is testing the content. Every platform has some quirks which force you to create art custom made for that platform (your other choice is to use the minimal subset of what they all support). If you want a good looking game, your content has several versions, so now, things like intersection and pathing code are slightly different between platforms. GPU behavior is different enough that using GPU's for anything results in multiple test code paths. AI isn't always fully deterministic, by design, making it even more difficult to test. So, what you have in the end are some deterministic tests to make sure your foundation is sound, and "fuzzy" tests to catch many problems in the higher level constructs, and your final line of defense are play testers, which is an awful job!
Now, we engineers know how to make the games stable and reliable, and our time estimates are frequently at odds with the PM's. You MUST make Christmas, which in the days that I worked, where CD's must be mastered, meant code freeze and asset freeze in October sometime. So, in this mad scramble to make a hard deadline, quality suffered.
It was much more important in the old days (PS1, PS2, GameCube, N64, etc), to get it right the first time, as you couldn't ship patches, but the notion of patching games allowed release versions to be more buggy, since people kicked the can down the road. Granted, the earlier consoles only supported simpler games, and the diffiulty was dealing with the particulars of the system, not the game itself. The PS1 had no Z-Buffer, the PS2 had a CPU/GPU combo built by a crazy person with an PS1 as its IO controller, and the PS3 required you to write DMA management code, while the GameCube had a memory model that was nuts and matrices were only 4x3 so you couldn't do projections. So, you spent most of your time dealing with these quirks and the game ended up simpler due to the effort being spent elsewhere.
Now, it's much easier, consoles are effectively PC's with powerful general purpose CPU's and GPU's, so it should be possible to write high quality, reusable engines.
I mean the biggest issue with creating good bots is getting the required data to make decisions such as "you're getting attacked, you need to use this ability" etc, but if you're actually the company writing the game, there shouldn't be any issues creating APIs for that. Or even going directly into memory and read it out, you don't need to worry about getting banned after all.
Imagine Integrationtests in the form of bot scenarios. Is that too much overhead to implement in the usually very budged oriented setting of game development?
That said, it would be wise to assume that if you have not heard much about this and wondered why, the answer is probably that it’s harder than you think.
Having been a game lead for a decade, I can safely say that getting game telemetry into the test bots is not the hardest part of the job, it’s one of the easiest things to do. One example of something much more difficult would be testing the game while it changes. Don’t forget that writing a cheat bot for a game that’s done is nothing like testing games that are in development and changing every day.
Testing non-deterministic code paths are a really hard problem because you don’t really know what to test for.
Also, Unity games are usually written in C# (I think it's quite safe to say that - overall - most games are not written in C++ but in C#), yet I've seen no data so far which would indicate that Unity games have any less problems than games written in C++ during production and after release (if anything, the opposite seems to be true, not for technological reasons, but because Unity is so much more beginner-friendly).
I'm no fan of C++ either, but blaming a programming language for bugs and quality problems without any counter examples at hand is a bit ridiculous.
If we eliminate all games with less than 1000 sales or something, I think it would be a very low confidence estimate.
- there is very little information in the community on how to do this kind of engineering efficiently (at least I haven't encountered it nearly as much as I have when I transitioned to application development in higher level languages) - there is very little code sharing in the community and zero standardisation - everyone reinvents shit from standard library, coding conventions, what subset of the language is "allowed", etc. etc. - this means developing good tooling, practices and patterns across large projects is hard
Unity is written in C++, C# is scripting layer and more importantly I doubt Unity developers doing C++ are C# engineers with C# application development background where stuff like automated testing is pretty standard.
If you aren't a huge studio writing a AAA game or thereabouts in complexity you probably can't write a game engine efficiently.
The number of places that do have that scale and need help on how to be efficient is probably vanishingly small so not a lot of self help pops up.
> (at least I haven't encountered it nearly as much as I have when I transitioned to application development in higher level languages
Pick an engine and use its language whatever that is. Unity gives you C#. Unreal gives you a C++ dialet with some nice features and a blackboard system for scripting that is quite powerful.
> there is very little code sharing in the community and zero standardisation
From my experience there is little code sharing within a studio so expecting there to be code sharing across the entire community is hard.
Assuming you are looking for high performance (AAA) you basically have to write a bespoke engine for your gameplay. And there are a lot of different ways of doing gameplay. Heck whether you have loading zones or continuous loading makes a huge impact on how some fundamental things work (again if you aren't huge there are workarounds that make it easier to generalize in exchange for performance)
> everyone reinvents shit from standard library, coding conventions, what subset of the language is "allowed"
The standard library has some huge performance problems for video games. Mostly around allocation rules. Unsurprisingly algorithms for millions of things can have bad performance side effects on tens of things (and visa versa).
> this means developing good tooling, practices and patterns across large projects is hard
To be fair I think this is generally true. Games just tend to build up their codebases super fast but maintaining any huge code base is a pain in the best cases and it is never the best case.
Don't need to search, factorio used to use boost: https://factorio.com/blog/post/fff-223
In a larger studio, even if code sharing isn't common as is expressed here, at a minimum having someone around that sees boost and says "don't do that" is probably a given.
Not even close.
That is true in the most literal sense, but Unity have adopted a programming model that makes much of the normal tooling around .NET useless, and abhors automated testing.
Adopting good testing practices in Unity still largely revolves around developing parts of your game as .NET libraries that get tested before being built and integrated into the game, and it's not too surprising that many developers don't go down this route.
Boost has over 160 libraries and counting. I wouldn't recommend every one of them (some have wacky interfaces, slow compile times, and/or experimental designs), but many of them are excellent, and I don't think it's very difficult to tell them apart.
Regardless, I find your insinuation that Boost users are "stupid" to be extraordinarily uncharitable to library users and developers.
You'd be surprised. Here's a talk from Croteam on how they test their games and engine: https://m.youtube.com/watch?v=YGIvWT-NBHk
I'd wager all major engines are exhaustively tested.
Trouble is, there's combinatorial explosion of game state, user input, assets, scripted behaviour and engine, so there's a huge area to cover.
I'd wager the popular engines are well tested because of the number of titles shipped on them not because they have good testing automation - but TBH I haven't worked in this industry for almost 10 years so maybe things changed.
I believe the same is true of "best practice" today: game studios aren't actually behind the curve, you just don't hear much about how things are progressing on this end because most of the conference talks aren't about broad concerns like testing, they're about the myriad specialities of the field.
And there absolutely is a legacy-code thing that hinders AAA in many cases. When the engine is old, that's good, because it's shipped something, but it's bad, because it's using older practices and Things Have Moved On.
Oh and I spent about 3 years working with UE4. The engine is a bug fest.
Unit tests etc aren't that useful in games so no you won't find much of that stuff.
With everything else being equal Unit tests are as useful for games and game engines as they are for any other type of software. If you care about your software being correct (or let's say less incorrect) you need to test it. And preferably you write automated tests. Games (and especially game engines) aren't really by default an exception to this.
(Of course you can just say that this game is not worth the effort of getting better quality so we don't care if it's a bug fest and that's fine).
Getting a game(or engine) into a particular initial state so you can make the unit test even work would be a massive pain the ass.
Heavy assertions are more useful as they test actual running code, and they are executed every time the program is run. You can still write "unit test" like code using assertions, but having some code that only executes in the development build on startup to check your math library or whatnot.
Not defending Unreal, I don't particularly like that engine, but that has more to do with the codes age and bloat.
It's not hard and definitely not impossible. Just requires a mind set towards quality and testing.
1) you can only test-driven-develop so much in games, and that line usually stops at the engine level because the game itself is in flux so much. Automated testing is confined to making sure that checkins build on every platform. Game dev engineering is fundamentally different than programming in other fields because the goal posts move constantly.
2) If an engineer writes code expecting there to be no more than 1024 physics objects in the system, tells the designers and artists this, but then they turn around put in 2000 colliding pieces of silverware on a dinner table "because it needs to feel like a big feast" is that an engineering problem, or an art and asset management problem? Because something like 80% of my bugs are shit like this.
3) Professional game codebases use their own styles (i hesitate to say dialects) of C++ that do the things we need them to do the ways we need to do them. We don't use anybody else's framework; all of the bonus stuff we're doing lives in macros that can be inspected if an issue arises. But, please don't push your language purism on anybody else. What a tired argument to have.
> Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had.
It is hard to describe because it sounds the same. But game development is really different because there are no fundamentals of things you care to test.
Even doing a combat sequence requires a herculean effort. You need basically the whole game running because otherwise what is the point. You fake out most of the data so your test doesn't fail when the designers decide to make combat harder. But now it ends up that pressing A defends instead of attacks because reasons and all of your combat tests now fail.
This is all fixable but it makes the cost per test of anything but the tiniest things hard enough that broad test coverage can sometimes be a determent as you end up testing what the game is now which means it will all be thrown away if your assumptions change.
Sure that can always be the case but "the sum of the lines equals the total" kind of tests are much less likely to backfire in this way.
> Plenty of teams have idiosyncrasies around their tool chain
C++ is chosen because the tooling doesn't exist outside of C++. Full stop. You have to build all the tooling in language X which when talking about 3D graphics is a huge amount of tooling.
Rust is starting to have some cool stuff but if you compare you will see there is a world of difference.
Thus if you choose not C++ you get to write C wrappers around your API and deal with all that nonsense since C++ interop is the worst in most languages.
At some point Rust will get proper C++ interop and then the gap will be smaller but for now you are giving up a ton for a slightly safer language by not choosing C++.
Also note that nearly everybody writes a huge amount of non-C++ code, they just call it a scripting language instead.
It always turns out that the game has a trivial state-of-the-art-circa-1980 design with a very small featureset. Nobody doing this is also writing a large RPG or even a Mario style platformer.
So, you can do it, but you spend a massive amount of "scope points" doing it.
The language tooling is much the same way. To do games - big or small - you need lots of I/O handling, and this immediately leads you towards talking to the OS directly, which leads you towards either C or C++ because that's where the tools and resources are. You can get a binding of SDL or whatever for your language, but that's effectively limiting the scope of your engagement with I/O - if the framework doesn't work, you have to debug it across a binding layer which is always iffy. And it can really hurt when talking about console dev.
I’m attending a computer graphics course and have to write a minecraft clone - and I don’t yet “feel” the performance of the CPU vs the GPU and sometimes I have trouble deciding which one to push.
Like, am I allowed to call glDraw* multiple times per different objects when the data is already on the gpu, or should I try to push somewhat dissimilar objects into the same buffer and write more complicated shaders differentiating them (without ifs if possible?) or are gpus so performant nowadays that unless I do something stupid I should not worry about it?
An example of this is the Swedish game dev company Embark Studios who, after a significant amount of effort, managed to get NVIDIA PhysX (which is widely used in AAA games) working with Rust [1].
Tooling for 3D modeling/texturing/rigging/etc is significantly more complex and powerful than it was 20 years ago, yet Pixar doesn’t need fewer artists for a movie today compared to Toy Story - in fact quite the opposite.
AI techniques useful to artists will get folded in the tooling and enable artists to make even more detailed/complex games & movies, but that doesn’t mean the AAA games of 2030 will require fewer artists.
However, talented small teams will likely be able to leverage them to create things that would have been inconceivable from a small team a decade ago.
The idea was that your small indie team could keep up with the big content demand since you had algorithms generating seemingly evergreen content for your players.
What happens in practice is that the players begin playing your game at a meta level, learning how the generative process itself works; effectively removing the benefit of procgen content. As an example, consider Spelunky which claims it can generate millions of unique caverns to explore. If you assess that claim visually, it's true. But watch the streamers play and you will see that they 'speak' the algorithm. By observing the shape of a cavern in one area of a map, they know the algorithm had to make a specific concession elsewhere in the map. So this content isn't really procedural for them anymore.
Even if the "AI" content generators get more intelligent, it won't free up resources. Mechanically interesting content, as demonstrated by the indie games of the last decade, is just a different form of handmade content. A designer hand made the procgen algorithms. Players ultimately bond with the designer(s), no matter what meta level they build the content at. In a hand-built game you might start to get a sense for where the designers hide treasure chests. In a procgen game you learn the algorithms themselves and how to predict and abuse them.
There's another form of game content, story content, which remains an edge for humans. Algorithms, or "AI" if you must, can't compete here. It would be the same as waiting for AI that can write award winning movie scripts.
The marriage of story content and mechanical content is a superior game-making formula to the procgen approach. So the _some point_ you refer to is probaby far away still.
If you want to calculate how much "AI generation" or "procgen" is freeing up resources you also want to look at how long people are playing these games for and how many resources were used in making them. If indie teams of 2 or 3 can collectively make the world play their games for longer than it plays games made by much bigger teams, then that's a definite freeing up of resources. And this is what's happening today to some extent.
Pedantically, wouldn't that be increasing the consumption of resources defined as man-hours?
> Pedantically, wouldn't that be increasing the consumption of resources defined as man-hours?
Only if you remove the distinction between paid employee man-hours and consumer man-hours (which, economically, is an externality, yes, but generally treated as a positive one, called 'engagement').
The best, most enduring games all seem to have procgen, randomization, or a "construction kit" for slower procgen: player-made levels and curation.
I think it's right to be ambitious in the long term though. Procedurally generated levels in 10-20 years are going to be wildly more advanced (and satisfying, and unpredictable) than now. I also wouldn't bet against award winning movie scripts in my or my kids' lifetimes.
Humans aren't really good at making an area feel wild or natural... we stink up the place with hidden order. So algorithms that generate terrain, tree placement, tree shape, etc are already outperforming humans because their form of pseudorandom is more natural-feeling than our own.
I don't know how famiar you are with the industry so apologies if I'm over-explaining, but check out SpeedTree if you want to see one of these products. It's really cool.
Naturally, one may wonder if this has resulted in the labor market disappearing for artists. But it hasn't. The demand for high quality art skyrocketed because the tools economically allowed for it. (In some ways the market for artists has gotten smaller, because there is less appetite for low-skill Photoshop monkeys, but highly skilled technical artists are much more in demand, because we are starting to see dramatic differences in productivity between artists just like we have seen in engineering.)
So, yes, we will see AI in content generation, but it isn't going to be in the form of replacing art bottlenecks. It will manifest in new tooling for artists, who will become more productive, and there will be even fewer artists who have good mastery of these tools, and the demand of quality will increase further. Which would lead to similar bottlenecks to today, though I believe artists will be paid better, and it will become even tougher to break in.
But are the artists treated better than they were a decade ago?
The wife of a good friend of mine is a digital artist. She works on films. The pay in her field is crap, as are the working conditions - meanwhile, competition for paying jobs is fierce.
When computers became an order of magnitude cheaper, the demand for, and the pay rate for computer programmers sky-rocketed. It doesn't seem like the same thing happened with artists.
The pay is OK but not fantastic: generally in the 60-80k range, depending on experience. Many of the very low paying jobs ($15/hr types) have disappeared. And I do know decent artists that frequently get recruiter calls, albeit usually for a couple large poorly managed studios that have trouble filling positions.
I think it depends a lot on the exact nature of the work being done as an artist. If it's more technical, it seems to demand better wages, but in games there are still some fairly nontechnical roles that are either paid poorly or farmed out to offshore agencies.
Work conditions still aren't great, but this is mostly at bigger studios.
This has already happened. AAA is now a small and ever-shrinking fraction of the overall games market.
Will raytracing the best engine that a 30-person team can build in 2 years any simpler? No, because "the best engine that a 30-person team can build in 2 years" is at a particular level of complexity by definition. Will it make the gap between the best engine that a 30-person team can build in 2 years and the best engine that one guy in his bedroom can build much smaller? Yes, yes it will.
Custom game engines are already dinosaurs; most of the money in games is elsewhere. I'm sure they'll continue to be produced, just like you can still pay a lot of money for a mainframe today. Mainframes were never defeated, not exactly - they can still do things that commodity hardware can't do. But they just became irrelevant.
This entirely depends on how you define AAA.
I don't know how you can know this. No one has:
- a widely agreed-upon definition of AAA (budget? team size? total work-hours that went into it?)
- an estimate of how much revenue AAA games generate from subscriptions, DLC, and any other add-ons
Without both of those, how can you even guess at AAA games' collective market share? Citation definitely needed.
> Custom game engines are already dinosaurs; most of the money in games is elsewhere. I'm sure they'll continue to be produced, just like you can still pay a lot of money for a mainframe today.
What does raytracing have to do with the popularity of custom engines?
Whether I'm planning to build a custom engine or license an existing engine, aren't we discussing whether my job is easier with raytracing than without it?
Put another way: it seems like the article is comparing reusable engines without raytracing to reusable engines without raytracing, as well as comparing one-off engines without raytracting to one-off engines with raytracing.
I don't think being able to use raytracing is going to move any dev team from one option to the other. If they had the desire and resources to build a custom engine, they'll probably still do that (against all logic).
> Mainframes were never defeated, not exactly - they can still do things that commodity hardware can't do. But they just became irrelevant.
Off topic, but this is not a good analogy. The "cloud" is just a network of mainframes, which means mainframes are arguably the dominant form of computing (and will become more dominant as thin clients, like Stadia, rise in popularity).
Mainframes were not necessarily purpose-built, unique machines. They were instead defined by their sharing model, which is where the term "personal computing" comes from -- as a contrast to the mainframe/server model.
Raytracing is being used as a synecdoche for technological advances in rendering. The point is that as off-the-shelf rendering improves, the advantages of a vertically integrated engine diminish.
> Off topic, but this is not a good analogy. The "cloud" is just a network of mainframes, which means mainframes are arguably the dominant form of computing (and will become more dominant as thin clients, like Stadia, rise in popularity).
> Mainframes were not necessarily purpose-built, unique machines. They were instead defined by their sharing model, which is where the term "personal computing" comes from -- as a contrast to the mainframe/server model.
That's a very essentialist perspective; there are a lot of ways mainframe computing differs from typical personal computing and no single one of them is definitive. As a programmer, a mainframe offered you a reliable computing environment, because they're built to be highly available from the hardware up. That's very much not the case with the cloud, which follows a worse-is-better approach where your jobs can be terminated whenever and you're expected to handle it. While shared computing resources may be making a comeback, I don't think that kind of ground-up high availability will, so while the cloud has some aspects in common with mainframes, it's very much not the same thing.
Not sure if it's true. Look at the profits of GTA, Cyberpunk, Witcher.
If anything that tells me they may have been better off investing that developer time into the game instead of a custom engine.
It's interesting you say that because a lot of the biggest games that get released are on custom engines. I'm most familiar with FPSes, but as I understand it, the cod games, the battlefield games, destiny, the halo games, cs-go, valorant, cyberpunk etc. are all on "custom" engines -- i.e. engines that are not provided commercially by third parties. I'm actually having trouble thinking of a big name fps that is on unreal or unity.
(I don't know how big these games are compared to mobile games, but I can say for sure that they are big enough that studios seem to be investing more and more on them, rather than less as you would expect if they were economically insignificant.)
p.s.: I share your opinion
That said, tere are a decent set of AAA games that license Unreal (PUBG, State of Decay, Borderlands series, Arkham series, XCom series) but it's rare among the absolute biggest titles ("AAAA" if you will.)
* Red Dead Redemption 2
* Call of Duty: Black Ops 4
* NBA 2K19
* Madden NFL 19
* Super Smash Bros. Ultimate*
* Marvel’s Spider-Man
* Far Cry 5
* God of War 2018
* Monster Hunter: World
* Assassin’s Creed: Odyssey
Further, what I would call the poster triplets for ray tracing Control, Metro Exodus, and CP 2077 all use in-house engines.
Sure - those are the AAA games we're talking about (and even then, a lot of those are long-running franchises that reuse a significant chunk of engine code between multiple installments). The biggest sales numbers will keep coming from that segment for a long time. But it's a shrinking segment in terms of overall revenue (much of which is no longer up-front sales), and even more so if you look at profit; each year the line where it makes sense to use a custom engine gets higher, and a bigger share of money coming in is from games built on commodity engines.
We're definitely in the hybrid zone for this console generation, though PS5 lead Mark Cerny said he'd been surprised to see some full raytracing (surely tech demos). Maybe PC's can do it, esp with 3090 and following years', but AAA seems mostly console-first (though e.g. cyberpunk 2077 is PC-first). Cross-generation games still need the old workflow, so you don't save until a whole studio can go fully next-gen exclusive... so, perhaps 2-3 years after PS6 gen launch in 7 years: 2030. Or the gen after that.
Couple of boundary counterpoints: most game engines aren't written in assembly despite it being faster; most CGI movies are fully raytraced (though looking at Soul, they seemed to chose a less realistic Special World that was easier to render).
Interestingly my wife said the same thing when watching it but I think the opposite is true. Everything in the great before is volumetric, so not easy to render at all.
As for cinematographers or some other change...it's all the same. Studios already think about lighting and framing and everything else in the scene. This won't be a revolution in that aspect of game design.
And while true for now, I think the title is plain wrong in the longer term. There will come a time in which hardware is sufficiently powerful that one no longer needs these tricks to create top graphics. And physically based raytracing absolutely will simplify all of rendering to a couple core components when that time comes.
https://www.cambridgeincolour.com/tutorials/cameras-vs-human...
In earlier headsets it was more narrow field of view and maybe that made it feel more obvious when my eyes were moving.
It will take such a long time to get realistic VR, that we will have direct neural interfaces that are much simpler because they can skip the encoding/decoding.
\tanget And perhaps at higher semantic neurons, so we can just invoke the concept of "cool graphics" without actually having to simulate it...
- [0] testufo.com
The old 144hz goal was/is nice if you want to display 24fps (movies) content but games don't work like that. They display correctly without any judder/pulldown at whatever frame rate you want (as long as you have the hardware to run it at that rate)
Sounds like the beginning of a sales pitch for a 720 Hz display. :-)
If testufo was randomized with multiple repeating patterns and random offsets, it could be a much better test. Here you can accidentally bias yourself.
- https://www.youtube.com/watch?v=tV8P6T5tTYs - https://www.youtube.com/watch?v=OX31kZbAXsA
And here's how I can probably immediately convince you that I can as well: drag your cursor at a rapid speed across your screen back and forth (e.g. 2-3 times across the width per second). Focusing on one spot, do you see individual cursors with gaps of background in between? (I'm assuming yes.)
Now on 120 Hz the gaps will be half as big as on 60 Hz. On 240 Hz they are half as big once again, but they are nevertheless noticeable. The size of the gaps easily allows you to distinguish between 120 and 240 Hz.
Until I can drag my cursor across the screen and see nothing but a continuous moving object, we haven't hit diminishing returns on monitor refresh rate yet.
laughs in light fields
My point was that even if hardware gets 100x better we still have a difference between a AAA game that can render a giant open world and a AA game that will use smaller worlds like a space ships with loading screens for each room.
I deeply wish this were true. But it doesn't seem likely to me.
Games are real-time. And frame rate expectations are getting higher (60, 90, and even 120Hz) which means frame times are getting shorter (16ms, 11ms, 8ms). Rasterization is likely to always be faster than raytracing so the tradeoff will always be there.
Pixar movies look better and better ever year. The more compute they can throw at the problem the better pixels they can render. Their offline rendering only gets more complex and higher quality every year. It's so complex and expensive I'm genuinely afraid that small animation studios simply won't be able to compete soon.
Maybe raytracing will be "fast enough" that most gamedevs can use the default Unity/Unreal raytracer and call it a day. But imho the author is spot on that AAA will continue to deeply invest in highly complex, bespoke solutions to squeeze every last drop and provide competitive advantage.
There's an upper limit to what's useful - the real world. If our minds can't comprehend it, you've hit the upper limit of practical use. Once in game graphics become literally indistinguishable from real life then we'll start to plateau in terms of complexity of "tricks" and raw computing power will be able to catch up.
Maybe. Alternatively games won't actually want to render photorealistic and will want to render with varying types of stylized graphics. Is that easier or harder? Probably a little bit of both.
We actually are at a point where we can real-time render photorealistic scenes... for certain types of objects. Primarily static environments. Photogrammetry is basically cheating, but it is highly effective! Mandalorian is famously filmed on a virtual set and it's cool as fuck.
Graphics is moving rapidly into physics. We might be soon be able to render photorealistic scene descriptions. However simulating the virtual world we still have a long, long ways to go. By simulation I mean everything from the environment (ocean, snow, etc) to characters. We most definitely can not synthesize arbitrary virtual humans.
Will we someday see a movie that _looks_ like a live action movie but is completely virtual? Oof. Maybe? But even if we could, would we want to? I'm not sure.
The original lion King worked because it was hand drawn graphics, the story , the dance and expressions on cat faces worked because graphics was limited.
The new one was a dull imitation with no understanding of what made it successful.
you should checkout the screen rant video on this https://youtu.be/sbx97kKHUHw
Art and games have been successful even if medium is limiting,ppl still read books watch plays and enjoy playing games like FNAF even if far superior tech is available.
If the limitations of medium wouldn’t matter, people and studios wouldn’t pay billions of dollars for VFX.
The new Beauty and Beast movie however was a whole lot worse than the cartoon. Lumiere, Cogsworth, Mrs Potts, etc all felt a lot more human and emotional in the cartoon than with the “realistic” CGI. They actually looked creepy in the new movie.
Is that to let them create something like a (say) 16K archival master that covers all the formats they might want for the foreseeable future without an expensive re-render?
Way too slow for VR, try 5 ms just to be sure.
If they get it better and more relevant in the future we can check again!
[1] Few as in "not enough to matter much". There can still be millions of enthusiasts, but the market predictions and "disruption" didn't pan out.
"Disruption" means that you create a less qualitative, but cheaper product that would cannibalize the margins of the product of a company whose products are high-margin (think cutting-edge CPUs of Intel). This "cheap" product will for a long time ridiculed as toy (but nevertheless sell in huge amounts because it is cheap) - until it suddenly "lifts off", becomes an inconvenient competitor, and disrupts the market.
Does this description of disruption sound like any market prediction from the last 10 years how the market for VR glasses will develop? I don't think so.
Not exclusively. That's disruption in the sense of "undercutting".
Disruption in the large also means:
"radical change to an existing industry or market due to technological innovation"
E.g. "things will change in how we do IT, entertainment, education etc due to VR".
That's because the limiting factor on VR hardware value to the consumer isn't resolution, or scene complexity as measured in polygons, or FPS, or any of the other metrics by which desktop games and game hardware are evaluated. It is latency. Latency in estimating and rendering the user's pose, movement, and viewport changes.
We would have had widespread adoption of immersive content by now if VR experience developers would be willing to sacrifice the 'traditional' metrics in favor of a ruthless focus on latency (and ideally push the hardware vendors in that direction too), but instead everyone keeps chasing "AAA graphics in a headset" , which, given that AAA graphics are (as the OP states) an ever-rising bar, can't ever really work except on whatever the current highest-end hardware is, and any experience that breaks people's willing suspension of disbelief on everything but the highest end hardware can't end up being a commercial success, any more than a multiplayer game with noticeable lag on anything under gigabit FTTH could (and for the same reason, except that the lag for VR starts with the single-player version, and only gets worse from there).
But the fact is it brings an entirely new genre of games that can simply cannot be played on a keyboard/mouse/joystick & monitor.
However you seem like someone who enjoys pretending like VR is a little fad and doesn't offer anything new to the game space- so whatever.
Whether it's absurd, tons of pundits still did tout it to high heavens, big media outlets, tech startup sites, etc. The usual BS propping up they do, like they did for stuff ranging from fuel cells to grid computing and autonomous cars at different times.
"The VR revolution", "The next big thing", "How VR will disrupt everything", "revolutionize the economy", and so on. This started around 2012 and peaked around 2016 or so.
>However you seem like someone who enjoys pretending like VR is a little fad and doesn't offer anything new to the game space- so whatever.
"Offering something new to the game space" is a spectacularly low bar compared to the oversold promise of consumer adoption and applications of VR.
And even at that, VR thus far didn't even make a real dent in the gaming space, now that you've mentioned it. So much for all the 2012-2017 hype of the tech finally being "nailed".
It's doubly funny to me, because I have memories of the first VR fad, in the 90s. At least then besides the media hype, we've also got a few camp movies out of it ("Lawnmower man" and co).
They just don't have close ups of humans...
https://www.starwars.com/news/the-mandalorian-stagecraft-fea...
No I think they literally do photograph the LED screen on the stage with the actors and that is the final composite. Just like with traditional back- and front-projection.
I guess it’s graded etc, but so is a completely natural scene.
Backgrounds are, by definition, not what the viewer is going to focus on. Oftentimes they'll even be out of focus and therefore slightly blurred.
Don't get me wrong, Disney's interactive set technique is revolutionary and shows how far Unreal has come, but I still expect future Pixar movies to take hours of rendering for every single frame (because by now that's basically Pixar's selling point).
It’s very common for film CG frames to take a few hours but not usually 12 hours. You need to be able to start rendering jobs at the end of the day and have them ready for dailies on the morning. Jobs that take more than 4 hours risk clogging the queue and taking 2 days per iteration rather than 1, so people are motivated to keep it reasonable.
Another important difference between games and film is antialiasing techniques. Much of the difference in render time between them is in using fancier texture sampling and using lots more samples than games are willing to use or even need for decent quality.
Text rendering used to involve complicated hinting and then subpixel rendering.
These days on Retina screens you don't need hinting or subpixel rendering.
You just render it like any set of curves, no longer any tricks about it being a font, because it's hi-res enough.
What rendering tricks are you suggesting are being added rather than being taken away, for fonts?
I think the author makes a good point at the high end, but Raytracing is going to simplify the crap out of whatever we have that's like Unity in a decade or two (hopefully not Unity).
So many issues like transparency sorting, AO & GI, depth of field, baked and real time shadows and reflections you basically don't have to deal with the complexities of as an end user.
But you'll be leaving a lot of performance on the table with a one size fits all solution.
But even though parallelism can be increased, the gains from them are limited as per Amdahl.
Amdahl basically does not apply to graphics tasks which are embarrassingly parallel. Raytracing will be no different.
And in fact you're getting rid of many preparation steps pre render that may be tricky to parallelize well, and replacing it with brute force.
Ray tracing is good for rendering surfaces, reflection and refraction, but what we see is not limited to these things.
Volumetric stuff is all over the place: fire and other plasma, smoke and other aerosols, other forms of mixed-state matter. Also we have lots of liquid water on our planet, when it interacts with solids things and/or air, the visual complexity of the scene simply explodes. Look at boiling water, or streams of water, or shoreline. And to complicate things even more, some visuals depend on wave nature of the light: soap bubble, rainbow, or DVD disk.
Here’s some 15 years old tech.demo: https://www.youtube.com/watch?v=HsWh66MvqBg
Stuff like the brick wall with these shadows and reflections is awesome fit for RT. The final scene also good. However, close up scenes of the ground with water pouring all over the stuff, or that glass window with drippling water at 1:30-1:45 — RT won’t help a bit. And the rain / water / wet surfaces is about half of their shaders: https://developer.amd.com/wordpress/media/2012/10/ToyShop-Eu...
Hybrid render pipelines and AI enhanced RT passes is the future.
Where the traditional rendering code is all based on hacks and tricks that simulate real life, raytracing is simply a set of formulas borrowed from a physics textbook. An engine casting more realistic shadows than a ten million lines of code rendering library can take less than a thousand lines of code.
The current raytracing acceleration is intended to boost some highlights and shadows in practice. The hardware isn't powerful enough or optimised for rendering real, full-screen games. Even a 3090 will have trouble rendering some Minecraft scenes, for example.
If the right hardware ever becomes available for the right price with enough consumers, proper raytracing engines will be feasible and complicated render paths will eventually be much simplified. Perhaps not to the point that they can be, no, but they'd be much simpler than the engines we're using now. We won't be getting Disney levels of quality, even at our own computers' resolution, but we don't need that. Equivalent to today's graphics but with proper lighting everywhere is an amazing graphical advancement already. Time will tell if that will ever happen.
Common things that make a scene too difficult to handle for brute force sampling approaches: caustics, difficult to reach light sources (small or behind openings), many light sources, camera or motion blur, volumetric materials, etc. Algorithms approximating the behavior of real materials also is a complex topic, e.g. skin or hair are handled separately from dielectric or metallic surface models. Some optimizations can solve one problem well in isolation but fail when encountering a combination of them.
Here's one that tries to solve caustics and small light sources at the same time: https://cgg.mff.cuni.cz/~jaroslav/papers/2012-vcm/2012-vcm-p...
A couple of days ago, while gaming with a few friends and noticing a transparency ordering issue in the game we played and explaining to them, how and why that happens I realized something:
The raytracing accelerators we now have at our disposal can also be used to implement proper primitive level transparency ordering on the GPU; even if you don't spawn secondary rays, just being able to do a sparse ray-triangle hit test through the whole scene building the render order list in situ.
It's also possible to fix transparency ordering in rasterization using OIT and there is some hardware support for it (ROV), but it's also expensive and can be hard to do right so many games choose rough sorting
Their big problems is trees, since leaves modeled as flat squares with their the actual leaf shapes in textures (that are completely transparent around the leaves).
"Trees are our biggest problem in ray tracing. No matter what we do they're by far the slowest thing to trace, and they're much worse than on rasterizer, and if you have any ideas of how to ray trace trees efficiently, we're very happy to hear them. We've asked a lot of very smart people and no one has come up with a good answer yet. You can choose between geometry and alpha testing and both are kinda crap." - It Just Works: Ray-Traced Reflections in 'Battlefield V', GDC 2019.
But even then, someone will think "if I just use some smoke and mirrors I can quite easily get twice the performance, and those spare resources I can use for better AI/larger worlds/whatever".
Ray tracing (and physically based rendering in general) already simplifies things. Content creators can use "natural" parameters for material/lights, and you don't need to insert fake lights to achieve realistic shadows, dynamic lights. That's in a way "simplifying" things, but in the end it just gives more realistic games for the same or higher effort. Until everyone can use ray tracing, it'll also add another layer of complexity becauuse you need to make a separate path for ray tracing, so level editors and content creators need to ensure everything looks good both in both cases.
To be precise, before the plough was invented, they didn't plough. They often used hoes instead [0]. Intuitively, the plough would have let farmers process a greater area in the same time. But whether they did work a larger area might have depended on other factors such as land ownership - e.g. who owned the plot adjacent to yours. The actual history of agriculture is quite fascinating [1].
the real cost is not the rendering engine, that's a choice. the real cost in AAA quality graphics are the assets.
The main point is that basic ray tracing has an algorithmic complexity that is logarithmic or cube root of the number of primitives (depending on your BVH), while basic rasterization has linear complexity, and that generally remains true even with many effective culling techniques.
There are hybrid techniques that try to combine the best of both, so nothing here is absolute, but at some point if you develop a competitive log complexity per pixel rasterizer with a BVH, eventually it basically becomes ray tracing, and you might as well go with the method that trivially gives you shadows and reflections and bounce lighting too.
I think what the author is trying to suggest, is that the cost of implementing Ray Tracing Real Time Rendering with its own Engine, giving specific performance and quality requirement still matters because it is a relatively cheap operation with respect to AAA Games budget. Vast majority of cost are in assets, and specifically Graphics Assets.
I am not sure I agree, because with the current trends I would have thought the incentive largely in flavours of moving to Unreal 5 over time. I think it is more of an economics, business, unit cost and TAM question more than a technical question.
( Stop putting stupid money / budget in rendering pipelines or graphics asset and start paying attention to Physics, Game-Play, StoryLine, NPC interaction, player psychology, in-game economics and god damn bug fix. )
I guess using words "slow" and "real time" in one sentence isn't best describing state of art :-)
Anyway, can anyone recommend some open source library that does that? No need for textures, just simple engine with light sources, coloured triangle vertices and transparency will be enough.
Anti-Alias Something?
Raytracing helps make realistic art possible. But it still takes a great artist to do it. There are so many subtleties to the way common surfaces reflect light that are hard to capture, even once you are given all the sliders you can dream of. My experience dabbling with Blender, was that it can be quite frustrating: you KNOW what you have set up doesn't look quite perfect yet, but you can't quite express WHY or what you need to change in the material definition to fix it. So you end up settling for "ok", meaning the resulting render looks pleasing -- but it still looks like a render.
But I guess that's why I'm not an artist.