So all of this processing happens for one frame? And this is going on at 60 Hz?
It doesn’t always happen at 60hz (which is 16ms total processing time for all the above plus what’s in the post). Some games run at 30hz(33ms), some run unlocked (varying times), and some can even run at 144Hz (7ms).
As an extreme example, VR headsets normally require rendering twice (once for each eye), and run at 90FPS, which is 11ms to do your entire game, and render it twice.
Furthermore, these aren’t normally just guidelines, these are caps. If your game runs at 60hz, and you miss your target by 1ms, on most hardware you’ll end up missing the hardware refresh which means either you get a temporary drop to 30hz, or you get tearing (half the old image and half the new image), neither of which are particularly pleasant.
[0]https://www.scan.co.uk/shop/computer-hardware/monitors/monit...
For one, game engines normally don't use OpenGL unless they have to. The GL drivers on PC tend to be very weak. High end games target Direct3D on Windows/Xbox and these days are moving to Vulkan/Metal, even on mobile.
On Windows the GL driver situation is so bad that Chrome translates GL to Direct3D. This obviously will impose some overhead and complicate the driver bug situation still further.
Even when games do target GL they tend to exploit lots of vendor specific extensions and be tested against very specific graphics card/driver combos to enable them to workaround bugs and performance cliffs. Does WebGL even expose driver specific extensions? I don't think it does.
So you are not going to be competing with native apps on the web anytime soon in this area (as with all other areas...)
If you read through the patch notes for your video card drivers, you'll probably notice that every major game release gets special support right in the driver.
NVIDIA has three advantages:
1. Most PC gamers have NVIDIA cards, so developers test primarily against NVIDIA cards.
2. NVIDIA has a boatload of driver developers to hack game-specific fixes and improvements right into their driver. They work around game bugs, rewrite slow shaders, and other stuff like that. I assume that's part of why the graphics driver is now well over a hundred megabytes.
3. NVIDIA sends developers to major game studios to optimize and add graphical effects to their games. For instance, volumetric lighting in Fallout 4 was added by an NVIDIA employee.
I think AMD does much of the same, but they just don't have the money to do it to the same extent. That means more breakage and slower fixes.
Sure it is! Not right now maybe, but the web standards people working on it would love to be a viable platform for AAA games. Every so often they tout a new WebGL port of a well-known game as the harbinger of things to come.
Then why are they implementing an API that's been unpopular for years and is now being phased out entirely (for AAA games)?
Mobile. They picked OpenGL ES 2.0 for WebGL because it had comprehensively won on mobile. Apple went with ES 2.0 and Android followed. It's taking a long time for the mobile industry to migrate to ES 3 (which would allow WebGL 2) but ES 2 has been a decent stable baseline for a good few years now. That's quite unusual, and very helpful, given how fast-moving everything in tech is.
[Edit to add: ES 2 is based on GL 3, which was the first version to add programmable shaders. That was a huge API change, and an admission that the D3D approach was better. It's barely 10 years old. So any "GL is unpopular" arguments based on the old fixed-function pipeline are a red herring, I think.]
Mobile was and is more important than either desktops or consoles, because the mobile market is huge and growing, while desktops and console are at best stable.
Google came up with a clever technical solution (the ANGLE library) to emulate ES 2.0 on top of Direct3D, so that sidesteps the technical problems of OpenGL on Windows.
Now, for AAA games, desktops and consoles are obviously far more important. I think there are two responses to that:
First, a bet that mobile will gradually catch up and become equally important. There are a lot of factors involved, but on raw technical terms it's not such a bad bet. Mobile hardware iterates very fast, and some mobile CPUs are getting very competitive with desktops (recent iPads and iPhones especially). Sustained performance is an issue, as mobile devices have much stricter thermal limits; but you can put the same mobile SoC in a bigger box, like the TV set-top boxes that Google, Apple, Amazon and others are experimenting with.
Second, there's no reason WebGL 3 couldn't be based on Vulkan. WebGL 2 hasn't even been fully adopted yet, so it would obviously take a number of years to make that happen. Maybe desktops and consoles will have moved on to something newer and better by then, but maybe they won't.
The big question is whether mobile+web is catching up on desktop+console, or if it'll always be a generation or two behind. I think you'd have to be pretty brave to bet against them ever catching up.
I think they're more concerned with the politics of it than the technical requirements of that userset.
I'm sure politics plays into it, but for WebGL specifically, it must have been a pretty easy technical decision. Do you pick the 3D standard used on Windows, or do you pick the lower-end one used by iOS and Android (and can be made to work on Windows)?
You could ask why GL rather than D3D won on mobile in the first place. For that you have to look at Microsoft and ask why Windows Mobile failed (in all its different versions). I don't think you can blame that entirely on politics.
Games are heavy users of driver and card specific extensions for instance. But that'd be at odds with the web's portability commitments.
That's a good point. From a web standards standpoint, it's a very tough conflict to resolve. I think the web people are pushing for common standards. That takes time and it can get very political, but I don't see a better solution. And if they can get it right, portability is a good thing! I don't see why that necessarily means you'll always be behind the curve on performance. A more standardized, portable system can catch up via economies of scale -- it might be easier to learn, have better tooling, a bigger potential market, etc.
Second, there's no reason WebGL 3 couldn't be based on Vulkan
Vulkan is a very low level API designed for ultra-high performance use in engines written by professional engine teams, like Unreal. It requires the developer to write large quantities of code to even render a single triangle because you have to take manual control over the GPUs low level details. To a large extent it's preferred to GL because of better interaction with multi-threading.
It'd make no technical sense to try and expose a low level hardware-oriented API designed for multi-threading to a slow single-threaded language like JavaScript.
Do you pick the 3D standard used on Windows, or do you pick the lower-end one used by iOS and Android (and can be made to work on Windows)?
Somehow C++ does not have this problem. So how about: don't pick, expose all of them and let the developer use whichever is more appropriate?
You say you don't see any alternative to how WebGL handles driver extensions. Of course there are alternatives: just expose them all. Let there be vendor specific and proprietary stuff in web apps. Just because this is considered politically unacceptable by the ideologues who control the web platform does not mean it's actually unthinkable.
But that'd be against how the web is "designed" (browser makers cabalistically picking winners).
So in fact, I will continue to bet against the web and against mobile. People have been predicting total domination of iPhone/iPad since the day they were launched. I'll still be playing AAA games on consoles or high end Windows PCs 10 years from now, I'm sure of it.
It'd make no technical sense to try and expose a low level hardware-oriented API designed for multi-threading to a slow single-threaded language like JavaScript.
Web Assembly will be mature soon, and on the timescales we’re speculating about it could well have some form of threading.
Let there be vendor specific and proprietary stuff in web apps.
Unfortunately that would be a security nightmare. The Flash and Java plugins are good examples.
Some of the bottlenecks in WebGL are:
- Javascript. Modern JS engines are great, so the overall speed can be good, but you're still missing things like 64-bit ints, SIMD and threads. (Do game engines make heavy use of threads though? I don't know, but many influential games programmers seem to be wary of them!)
- The WebGL -> OpenGL translation layer takes some time. In Chrome it sanity checks your input (which you definitely want in a browser! GPU drivers are very insecure) and executes it in a separate process. Not as expensive as you might think, especially if you minimize your draw calls, which is good practice anyway.
- WebGL is basically OpenGL ES 2.0 (and WebGL 2 is ES 3.0), which is missing some useful features of full OpenGL, and doesn't offer the low-level, low-overhead access of Vulkan or Metal.
Depending on what you're doing, I'd guess WebGL might be no more than 2x slower (assuming plenty of optimization work). Some fiddly things might be 10x slower, or just not possible at all within the WebGL API.
WebGL 2 has a lot of very important new features, but support for that still seems to be patchy, and it's only just catching up with mainstream mobile graphics. It's a generation behind Vulkan.
Apart from that, an AAA game will have massive amounts of graphical and audio assets. Delivering that over the network is a pain and HTML5 caching is a pain. Doable, but hardly comparable to just loading it from local storage.
Oh, and one more thing! A big game wants sound as well as graphics, and WebAudio is a mess. And audio mixing is typically done in a background thread, so that's one area where the lack of threads in JS is a real problem.
Overall WebGL is very nice, it was a great choice to follow OpenGL ES closely (security problems aside).
Boy do they, yes.
I'm specifically curious about CPU-intensive stuff that is sharded out to multiple threads. That's your classic multithreaded programming, the sort of thing you'd do in scientific computing, but I always got the impression games people were skeptical about it, due to unpredictable performance and the high risk of bugs.
That way if you're running a multithreaded physics simulation you can get two separate bullet collision detections on one object, and instead of trying to delete the object twice you put both "kill this thing" actions on a to-do list. When it comes time to handle that in the main loop, you sweep through it for conflicts before executing any of the changes.
UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that. But here's an example (without much technical detail) of someone implementing multithreaded pathfinding for UE4: https://forums.unrealengine.com/community/work-in-progress/1...
Yeah, that's the kind of thing I was thinking of when I said games programmers seemed skeptical of threads.
Some of that coarse-grained parallelism could possibly be done in JS with Web Workers, but those have their own problems. (See the recent discussion here about the "tasklets" proposal: https://news.ycombinator.com/item?id=15511519)
Execution: You can google around for "game engine job system", but unfortunately, game engine blogs have really fallen off a while back as most of that crew has moved to twitter. So, the best material out there is in the form of GDC presentations such as "Parallelizing the Naughty Dog engine using fibers", " Destiny's Multithreaded Rendering Architecture", "Multithreading the Entire Destiny Engine", "Killzone Shadow Fall: Threading the Entity Update on PS4". Slides and videos are available around the web.
I never got to program one but I remember being fascinated by the weird architectures of the PS2 and PS3.
It occurs to me that there are some similarly weird architectures in mobile right now -- it's not uncommon to see Android flagship phones with 8(!) cores, which is just ludicrous. And there are lots of asymmetric "big.LITTLE" designs with a mix of high-speed and low-power cores.
Maybe I'm just reading the wrong blogs, but I've barely seen any discussion on how to optimize code for those crazy Android multicore CPUs, even though it seems like there's potentially a lot of upside. I guess Android is so fragmented and fast-moving that it's a tougher challenge than optimizing for two or three specific games consoles; also Android app prices are low so there probably isn't as much motivation.
Also, Apple is miles and miles ahead of everybody else in mobile performance, and they've consistently gone with just 2-3 cores. Their mobile CPUs and GPUs are very smartly designed, really well-balanced.
There’s also this talk - http://developer2.download.nvidia.com/assets/gameworks/downl... which talks about collision detection and collision resolution on the GPU that actually explains the architecture of the cpu engine quite well and shows what parts are serial and what parts are well parallelised.
Of course, you're still dealing with the event loop in most cases, which is probably a stumbling block when it comes to really low-level stuff. That said, there are even facilities for shared memory and atomics operations [3] these days, which helps. I've messed around with it a little bit on a side project- as a JS developer, it's really weird and fun to say "screw the event loop!" and just enter an endless synchronous loop. :D
[1] https://developer.mozilla.org/en-US/docs/Web/API/Worker [2] https://developer.mozilla.org/en-US/docs/Web/API/MessageChan... [3] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
(There are also older demos for asm.js + WebGL 1)
In general, WebGL and asm.js have done a great job of keeping up with the capabilities of whatever is the current-gen iPad.
Some additional reading on the topic:
https://developer.valvesoftware.com/wiki/Source_Multiplayer_...
https://blog.forrestthewoods.com/synchronous-rts-engines-and...
Since you're dealing with multithreaded CPUs + an asynchronous GPU, you can parallelize and sequentialize all this. How UE works in its DX11 and OpenGL renderers, on a 90hz game, is that you're gonna have the Game Thread (Physics/Gameplay) running for frame N, while the Render Thread (GPU commands / math ) runs for frame N-1, while the GPU executes frame N-2. This allows you a complete frame time of 33ms on a throughput of 90hz, at the cost of more latency.
I wonder what the cutoff is where someone starts to notice, but I imagine it's not much greater than 50-100ms
I have a gaming display that has a 1ms response time, if I play something fast and twitchy on my PS4 (Titanfall 2 for example) on my display and then on my tv I can 100% notice the difference between the two, even when you adjust the tv to compensate (gaming mode on tv): you really do notice the extra time taken for events to occur. Titanfall 2 is actually a pretty good example because of the insane gameplay speed. Here's some examples: https://gfycat.com/FrayedTameDamselfly https://gfycat.com/BreakableWealthyBlowfish https://www.youtube.com/watch?v=o7ARc-lxc2s https://gfycat.com/FrailSaltyDanishswedishfarmdog https://gfycat.com/JubilantNeighboringAmericancrocodile
Business apps have it a lot easier - the users are more forgiving, I guess either because they don't have a choice in the matter or any better options.
(Not to negate your point though. The crappy HR systems are painfully slow and incredibly frustrating to use, in comparison)
When programming with APIs like OpenGL or DirectX, you either know what you're doing and get everything right, or you get a completely black framebuffer and maybe some error codes. There's practically no in-between. For business applications you can get pretty convincing results just by throwing together some buttons and textboxes in NetBeans or Visual Studio.
If you had to look up CRUD vertex and texture data as database rows from another machine through the internet you wouldn't see a very smooth game.
It's all about structuring your data such that don't don't cache miss on your fetches and you don't stall your pipelines by doing operations that flush/occupy the same pipe.
You can easily see 10-50x performance boosts on the same dataset with the right approach.
Look up Data Oriented Design, Mike Acton talks about it a ton.
This is not a critique against either Design Patterns or Java, both of them are important and useful. And really, quite often you don't care about waiting 100 cycles, you have ten million times 100 cycles in every second. It's when you start crunching numbers in large quantities that it can get problematic. Modern CPUs are very fast, but only if you feed them data in the right order. Otherwise you're basically running on a machine from 20 years ago, just with 32GB of RAM. Best program design right now is to yes, do write your deep class hierarchies and whatnot, but also know how to identify the parts of your program doing heavy computation and how to restructure those so you can unleash the full power of a modern processor.
That said, in any performance critical application, you pay close attention to your memory hierarchy and you design and compile your application in a way to respect that as well as take advantage of the underlying hardware architecture and resources to the fullest.
This can actually be worse due to context switch overhead and caches you were using getting evicted by the now running thread hitting the same issue.
If you want to do this right you should be organizing your data so that the prefetcher and take advantage of it. All of those speculative things still have a cost.
A GTX 1060 has 4.4 TFLOPs of processing power. 1920 * 1080 = 2073600 pixels per frame, or 125 million pixels per second. Divide that into 4.4T and you get a budget of max 35,000 operations per pixel. That's quite a lot. In practice you will often spend time waiting for textures to arrive from RAM.
The techniques themselves have also been built up over decades. There's a lot of giants to stand on.
Edit: replaced 'driven' with 'oriented' - thanks Narishma
Or more: https://answers.unrealengine.com/questions/7459/question-is-... (person wondering if you can get more than 120 FPS in UE back in '14, one of the replies says they get "stat fps" in the 180s)
Unreal uses an architecture called deferred rendering[1]. It's more complex, but allows for a lot more tricks and the main benefit is that lighting performance is closer to constant, instead of increasing or decreasing with how many lights you have.
Forward rendering is the simpler alternative[2]. For Unreal, they had a forward renderer for mobile and were introducing one as an option for VR since the minimum frame time can be shorter.
[1] https://en.wikipedia.org/wiki/Deferred_shading
[2] https://gamedevelopment.tutsplus.com/articles/forward-render...