Clip control on the Apple GPU
rosenzweig.io
rosenzweig.io
It still staggers me that there is not 1 coherent reference for how to do all of this. Virtually every reference about clipping winds up with something like "and then the GPU waves its magic wand and everything is properly clipped & interpolated :D". Every paper I read has some "your real answer is in another paper" meme going on. I've got printouts of Blinn & Newell, Sutherland & Hodgman, et. al. littered all over my house right now. About 4 decades worth of materials.
Anyone who works on the internals of OGL or the GPU stack itself has the utmost respect from me. I cannot imagine working in that space full-time. About 3 hours of this per weekend is about all my brain can handle.
I'm not sure what issues you're hitting, but I've never found clipping to be that challenging or difficult. Also, clip control and clip space aren't really specifically about clipping -- clip space is just the output space of your vertex shader, and the standard "clip control" extension just controls whether the near plane is at 0 or -1. And 0 is the correct option.
I am not sure I understand why this would be rare. If I am intending to construct a rasterizer for a first-person shooter, clipping is essentially mandatory for all but the most trivial of camera arrangements.
Clipping deals with geometry that is partially inside the camera. Culling (either for backfaces or entire instances) is a preliminary performance optimization that can be performed in a variety of ways.
Clipping in X and Y is simpler at least, and again the guard band hopefully mostly covers you.
Yes. Guard band is an optimization that reduces the amount of potential clipping required. You still need to be able to clip for fundamental correctness.
If you totally reject a vertex for a triangle without determining precisely where it intersects the desired planes, you are effectively rejecting the entire triangle and creating yucky visual artifacts.
This simplified sample shows Sutherland-Hodgman with 4D clipping: https://web.archive.org/web/20040713023730/http://wwwx.cs.un... The main difference is the intersect method finds the intersection of a 4D line segment against a 4D plane.
In my experience it is most elegant to clip against the 6 planes of the view-frustrum in succession (one plane at a time). Preferably clipping against the near-plane first, as that reduces the set of triangles the most for subsequent clips.
Your triangles can turn into convex polygons after a clip. So it is convenient to start with a generic convex polygon vs. plane-clipping algorithm; The thing to be careful about here is that points can (and will) lie on the plane.
Use the plane equation (f(x,y,z)=ax+by+cz+d) to determine if a point is on one side, on the plane, or the other side.
It is convenient to use a "mask" to designate the side a point v=(x,y,z) is on. So: 1 := inside_plane (f(x,y,z)>eps) 2 := outside_plane (f(x,y,z)<-eps) 3 := on_plane (f(x,y,z)>=eps && f(x,y,z)<=eps) Let m(v_i) be the mask of v_i.
When you go through each edge of the convex polygon (v_i->v_{i+1}), you can check if you should clip the edge using the mask. I.e.:
if (m(v_i)&m(v_{i+1})==0 the points are on opposite side => clip [determine intersection point].
Since you are just clipping to the frustrum, just return a list of the points that are inside or on the plane (i.e. m(v_i)&1==1) and the added intersection points.
There are lots of potential for optimization, of course, but I wouldn't worry about that. There are lots of other places to optimize a software rasterizer with more potential, in my experience.
It was a pain to lean indeed and the best resources were quite old:
- "CLIPPING USING HOMOGENEOUS COORDINATES" by James F. Blinn and Martin E. Newell
- A Trip Down the Graphics Pipeline by Jim Blinn (yes the same Blinn that co-authored the paper above).
Yes, OpenGL on macOS is now implemented over Metal, but unfortunately a side effect of this is that implementation-level details that were critical to debugging and profiling OpenGL just no longer exist for tools to work with. Anything is possible? Maybe? I'm sure Apple Graphics engineers could make old tooling work with the new abstraction layer, but it's not happening.
Tooling investment is all on Metal now. But so much existing NON-LEGACY software relied on OpenGL.
So what do you do? You debug and perf test on Windows and Linux and hope that fixing issues there addresses concerns on macOS, and hopefully your problems aren't platform-specific.
This is how some graphics engineers, including myself, continue to ship for macOS while never touching it.
Edit: Also, Vulkan is a waste of time for anyone who isn't a large studio. No one wants to write this stuff. The most common argument is "You only write it once." No, you don't.
You have to support this stuff. If it were that easy, bgfx would have been written in a month and it would have been considered "done" afterwards.
The snarky response would be "you're not a graphics engineer if you're still using OpenGL" (please laugh at the bad joke)
But seriously, knowing how GPUs actually work, OpenGL is pretty horrible. Full OpenGL has 20 years of cruft hacked and glommed on to it. As someone who's job is to write OpenGL emulation it really needs to die and people need to stop using it.
If you don't care but want cross platform OpenGL ES then use ANGLE, or if you want something more modern that ditches most of that 20years of cruft look into one of the native WebGPU implementations (dawn, wgpu, ...)
> Also, Vulkan is a waste of time for anyone who isn't a large studio. No one wants to write this stuff. The most common argument is "You only write it once." No, you don't.
OpenGL is not API-machine precise, but widely this isn’t relevant for people who are primarily concerned with shipping products.
APIs like Vulkan, DirectX 12, and WebGPU need to exist, but there is too much fragmentation in the graphics API space now and because of this API decision making that has taken place, the situation will not be smoothed over for probably a decade.
Many people hoped for an OpenGL successor that provided API-machine precision without explicit verbosity that you could simply opt-into if you needed control. Instead what we ended up with were required definitions of API mechanics.
These aren’t obvious to implement. It also debatably moves engineering efforts to the wrong people. There are graphics programmers who should not be involved in adapter-device type programming, who instead should be focusing simply on graphics shader work and the like.
Vulkan is a textbook case of throwing the baby out with the bathwater in pursuit of a more precise API. It throws out development practicality by allowing vendor engineers to wipe their hands of things that some of us want them to take care of.
Imagine if every time you wanted to write software you first needed to probe for what CPU cores existed, and also define how to allocate memory, and also how your program should run.
This is needless. It would also create a scenario where people would say similar things like, “Well sure it’s verbose, but of course you would only write this once.”
And everyone, individually, would do this, because everyone, individually would be forced to, and they’d all have to maintain it.
And along the way, most would get it wrong. Or implement only a subset of desired behavior.
so if i understand correctly, it would have been better to have an api that had more of an onramp with increasing complexity when you needed it, but instead all we got was fragmented low-level apis?
The API is very antiquated, doesn't match modern GPU architectures at all and requires many workarounds in the driver to get the expected functionality, often coming at a performance cost.
Vulkan is nice, but it goes into the other extreme. It's very low level and designed for advanced users. Even getting anything on the screen in Vulkan is intimidating because you have to write everything from scratch. To go beyond hello world, you even have to write your own memory allocator (or use an existing opensource one) because you can only do a limited amount of memory allocations and you're expected to allocate a huge block of memory and suballocate it as needed by your application.
In comparison, DX12 is a bit easier to grasp. It has some nice abstractions such as commited resources, which take some of the pain away.
Personally I like Metal as an API. It is lower level than OpenGL, getting rid of most nasty OpenGL things (state machine, lack of pipeline state objects), yet it is very approachable and easy to transition to from DX11/OpenGL. I was happy when I saw WebGPU was based on Metal at first. WebGPU is my go-to 3D API at the moment, especially with projects like wgpu-native which make it usable on native platforms too (don't let the Web in WebGPU confuse you).
I honestly believe that this is the major reason. Developing a hobby project with OpenGL is little more than using SDL or GLFW to get a window with a GLContext and then you can just start calling commands. Vulkan is much more complicated and unless you're really pushing performance limits, you're not getting much of a benefit for the extra headache.
If you're doing anything advanced with OpenGL, yeah, debugging it is a pain and the statefulness isn't fun. But if all you're doing is trying to put some textured quads (two triangles) on the screen and you're not wrangling multiple framebuffers, there's not much to go wrong.
I assure you, none of the "love" for OpenGL comes from the elegance of its design.
Also RIP all the countless pessimistic "engineers" here and elsewhere saying we'd be waiting for years more for any graphics acceleration.
Edit: It is true though that AAA Gaming will wait: "Please temper your expectations: even with hardware documentation, an optimized Vulkan driver stack (with enough features to layer OpenGL 4.6 with Zink) requires over many years of full time work. At least for now, nobody is working on this driver full time. Reverse-engineering slows the process considerably. We won’t be playing AAA games any time soon."
Still, even if that be the case, accelerated desktop is an accelerated desktop, much sooner than many expected.
In any case, I don't think Asahi/M1 has proper KWin or Mutter support yet. It's still going to take a while before you get a truly smooth desktop Linux experience on those devices, but some hardware acceleration is definitely better than none!
Well... you were one of the most vocal critics saying it wouldn't happen anytime soon.
> unfinished, hacky implementation (the author's own concession)
Still more stable than Intel's official Arc drivers, so who defines "hacky"? ;)
> Plus, you're going to be stuck on x11 until any serious GPU drivers get written
Only because it is running on macOS, which supports X11 but not Wayland. On Linux, Wayland or X11 will both work, no problem.
> No MacOS-like trackpad gestures either, you'll be waiting for Wayland support to get that too
Again, Wayland will work on Day 1, it's just a limitation of running the driver on macOS until the kernel support is ready. When it is on Linux, Wayland will be a full-go.
Yep. Been beating that drum since 2020, looks history proved me right on this one.
> Still more stable than Intel's official Arc drivers, so who defines "hacky"? ;)
Apparently not me, I had no idea that the M1 supported Vulkan and DirectX 12.
Did you mean some other title? Even diablo 1 was a Windows game.
Most old games won't work because most are going to require OpenGL>=3. For D3D games from Windows, translation efforts have long been focused on Vulkan, and even then, you are very likely to require extensions which aren't present or further versions of OpenGL.
So that's to say that, it's not about AAA games, it's about Firefox, (full) KDE, and Minecraft. Which is going to take a few years, probably. There is no RIP, it's going about as well as those people were predicting - I'm expecting it's not going to be able to have full browser acceleration and play Minecraft until 2 more years at the very least.
These are the same in any closed source OS
https://help.apple.com/pdf/security/en_GB/apple-platform-sec...
This twitter thread is a perfect example of why it's needed
Bug fixes and performance work on the OS are even more likely to break your app than feature work is! Bug fixes in the OS are pretty likely to cause new bugs too…
OpenCL was created and open sourced by Apple.
>OpenCL was initially developed by Apple Inc., which holds trademark rights, and refined into an initial proposal in collaboration with technical teams at AMD, IBM, Qualcomm, Intel, and Nvidia.
OpenCL 1.0 released with Mac OS X Snow Leopard on August 28, 2009.
I'm talking about the GPU OpenCL implementation. Implementing OpenCL on the CPU is not really a big deal. I can guarantee you that these implementations were Intel and NVidia and were just redistributed.
Besides, rebuilding/redesigning the settings screen is a perfect "snow leopard" thing. That's not an actual Feature.
The problem isn't doing features, the probably is doing a bad job.
• Mountain Lion / Mavericks
• El Capitan
• High Sierra
Yes, these releases had new features, but so did Snow Leopard. Behind the scenes there was grand central dispatch, and in front there was QuickTime X, App Exposé, and Exchange support.
I also happen to think Snow Leopard is overrated, its legacy largely due to the fact that Lion was really bad.
This isn't to say Apple shouldn't do another polish release. They're overdue for one.
https://www.opengl.org/resources/libraries/glut/glut_downloa...
It's not unprecedented in the open-source space though; the PCSX2 PlayStation 2 emulator for example contains a MIPS -> x86 recompiler, and the RPCS3 PlayStation 3 emulator contains a Cell -> x86 recompiler.
Also though, since Rosetta 2 was released, it's had an incredibly slow implementation of x87 FPU operations, and anything that relies on x87 floating point math (including lots of games) is currently running about 100x slower than it ought to. Apple is aware of it but it's still not fixed in Ventura.
I hadn't heard of FEX before, looks interesting.
What software do people have x87-related issues with?
The author of REAPER has also written about it some: https://user.cockos.com/~deadbeef/index.php?article=842
There's been lots of discussion about the issue in the Codeweavers forums, and Codeweavers points the blame squarely at Apple, who have been, predictably, very quiet about it.
There's no macOS host to interact with on AWS Graviton3 instances.
Remember that this translation needs to happen at the graphics driver level. For fixed-function OpenGL where the application actually passes the graphics driver a projection matrix this would be doable. But if your application is using a version of OpenGL newer than 2004, the projection matrix is a part of your vertex shader. The graphics driver can't tell what part of your shader deals with projection, and definitely can't tell what uniforms it would need to tweak to modify the projection matrix -- many shaders might not even have a projection matrix.
"Neverball uses legacy “fixed function” OpenGL."
But also you could simply remap the Z coordinate of gl_Position at the end of the vertex stage, do the clipping in [0,1] range, then map it back to [-1,1] for gl_FragCoord at the start of the fragment stage.
Sure, it'd work for Neverball, but the article is clear that they're looking for a general solution: something that'd work not just for Neverball, but for all OpenGL applications, and would ideally let them give applications control over the clip-control bit through OpenGL/Vulkan extensions.
> But also you could simply remap the Z coordinate of gl_Position at the end of the vertex stage, do the clipping in [0,1] range, then map it back to [-1,1] for gl_FragCoord at the start of the fragment stage.
Yes, that was the current state-of-the-art before this article was written:
> As Metal uses the 0/1 clip space, implementing OpenGL on Metal requires emulating the -1/1 clip space by inserting extra instructions into the vertex shader to transform the Z coordinate. Although this emulation adds overhead, it works for ANGLE’s open source implementation of OpenGL ES on Metal.
> Like ANGLE, Apple’s OpenGL driver internally translates to Metal. Because Metal uses the 0 to 1 clip space, it should require this emulation code. Curiously, when we disassemble shaders compiled with their OpenGL implementation, we don’t see any such emulation. That means Apple’s GPU must support -1/1 clip spaces in addition to Metal’s preferred 0/1. The problem is figuring out how to use this other clip space.
This is the same as modifying a projection matrix -- you're doing the same post-multiply to the same column. But note that there's no guarantee there's ever a projection matrix. Clip space coordinates could be generated directly in the vertex shader.
Very few applications in the grand scheme of things use Vulkan, and a minority of games do.
Therefore the ROI on supporting OpenGL is very high.
Also, Vulkan isn't popular with game developers because availability sucks. Vulkan doesn't run on macOS. Or iOS. Or 40% of Android phones. Or Xbox. Or PlayStation. Or Nintendo Switch[1].
Unless you are targeting Windows (which has DirectX and OpenGL already), or those 60% of Android phones only, or Linux, why would you use Vulkan? On Windows, DirectX is a generally-superior alternative, and you get Xbox support basically free, and if you also support an older DirectX, much broader PC compatibility. On Android, just use OpenGL, and don't worry about separate implementations for the bifurcated Vulkan/OpenGL support. On Linux, just use Proton with an older DirectX. Whiz bang, no need for Vulkan whatsoever. Yes, some systems might perform better if you had a Vulkan over OpenGL, but is the cost worth it when you don't need it?
[1] Technically, Vulkan does exist for Nintendo Switch, but it is so slow almost no production game uses it, and it is widely considered not an option. Nintendo Switch is slow enough without Vulkan making it slower. Much easier just to use the proprietary NVIDIA library.
My understanding is that this hardware abstraction layer for mesa is way easier to implement than a full vulkan driver, especially since the earlier versions of OpenGL only require a small subset of the features that a vulkan driver requires.
It's an accident of history more than anything else. Once the reverse engineering is further along, I expect a Vulkan driver to be written for it, and the Gallium one to be phased out in favor of Zink.
> And there’s the Metal that Apple uses themselves, an internal API adding back features that Apple doesn’t want you using.
Apple does stuff like this so much and gets so little flak for it.
I use macOS since it seems like the least bad option of you want a Unix but also don't want to spend a lot of time on system management, but this is a real turn-off.
To be fair, Windows has a ludicrous amount of undocumented APIs for internal affairs as well, and you can get deep into the weeds very quickly, just ask the WINE Developers who have to reverse-engineer the havoc. There is no OS without Private APIs, but Windows is arguably the worst with more Private or Undocumented APIs than Apple.
This actually bears parallels to Metal. Until DirectX 12, Windows had no official way to get low-level. Vulkan and OpenGL are only 3rd-party supported, not Microsoft-supported, Microsoft officially only supports DirectX. If you want Vulkan/OpenGL, that's on your GPU vendor. If you wanted low-level until 12, you may have found yourself pulling some undocumented shenanigans. Apple hasn't gotten to their DirectX 12 yet, but they'll get there eventually.
As for why they are Private, there could be many reasons, not least of which that (in this case) Apple has a very complicated Display Controller design and is frequently changing those internal methods, which would break compatibility if third-party applications used them. Just ask Asahi about how the DCP changed considerably from 11.x to 13.x.
Can anyone in the know give more information here? Why would Apple want to do this? What could they be doing that's so complicated in the display controller?
and
https://twitter.com/marcan42/status/1415360411260493826?lang...
and
https://twitter.com/marcan42/status/1526104383519350785
As to why? Well, if it ain't broke don't fix it from iPhone, but it is still a bit of a mystery.
In a nutshell from those threads:
1. Apple's DCP silicon layout is actually massive, explaining the 1 external display limit
2. Apple implements half the DCP firmware on the main CPU and the other half on the coprocessor with RPC calls, which is hilariously complicated.
3. Apple's DCP firmware is versioned, with a different version for every macOS release. This is also why Asahi Linux currently uses a "macOS 12.3" shim, so they can focus on the macOS 12.3 DCP firmware in the driver, which will probably not work with the macOS 12.4+ DCP firmware or the macOS 12.2- firmware.
I can totally see why Apple doesn't want people using their low-level Metal implementation that deals with the mess yet.
Seems the DCP driver was originally all on the main CPU, and when apple got these cheap coprocessor cores, they took a lazy approach of just inserting a simple RPC layer in the middle. The complexity for Asahi comes from the fact that it's a c++ API that can change very dynamically from version to version.
And yes, these ARM coprocessor cores are cheap, apple have put at least 16 of them [1] on the M1, on top the 4 performance and 4 efficiency cores. They are an apple custom design that implement only the 64bit parts of the ARMv8 spec. I'm not entirely sure why the actual DCP is so big, but it's not because of the complex firmware. Potentially because the DCP includes enough dedicated RAM to store an entire framebuffer on-chip.
If so, they will be doing this because it allows for lower power consumption. The main DRAM could be put in a power-saving mode and kept there for seconds or even minutes at a time without having to wake it up multiple times per frame, even when just showing a static image.
It's freeing not to be beholden to the likes of someone like Tim Cook who, it would seem, spends the majority of his waking hours figuring out how to hide anticonsumer decisions under rugs.
It would be one thing if the private APIs were limited to system frameworks and features while Apple’s own apps weren’t allowed to use them, but they do. E.g. The Swift Playgrounds app for iPad is allowed to share and compile code, run separate processes, etc. which isn’t normally allowed in the AppStore. They also use blur and other graphical effects (outside of the background blur material and the SwiftUI blur modifier) that are unavailable outside of private APIs.
It stinks because of the perceived hypocrisy and the inability to compete on a level playing field or leave the AppStore (and I say this as someone who normally doesn’t mind the walled garden!)
The best known example of these methods is how Microsoft has exploited the replacement of MS-DOS with Windows 3.0 and especially with Windows 95.
During the MS-DOS years, the only Microsoft software products that were successful were their software development tools, i.e. compilers and interpreters, and even those had strong competition, mainly from Borland. Those MS products addressed only a small market and they could not provide large revenues. The most successful software products for MS-DOS were from many other companies.
That changed abruptly with the transition to various Windows versions, when the Microsoft developers started to have a huge advantage over those from any other company, both by being able to use undocumented internal APIs provided by the MS operating systems and also by knowing in advance the future documented APIs, before they were revealed to competitors.
Thus in a few years MS Office has transitioned from an irrelevant product, much inferior to the competition, to the dominant suite of office programs, which has eliminated all competitors and which has become the main source of revenue for MS.
Also, to Alyssa, if she's reading this: you're just going to have to implement support shader variants. Build your infrastructure for supporting them now. It's going to be far more helpful than just for clip control.
But yes, the Vulkan extension was just poorly specified, allowing you to change clip spaces between draws in the same render pass is, again, ludicrous, and the extension should just be renamed VK_EXT_i_hate_tilers (like so many others of their kind). Every app is going to set it at app init and forget it; the implementation using the render pass bit and flushing on change will cover the 100% case, and won't be slow at all.
I admittedly have zero experience with Mesa, but it seems like shader variants is something that should be common infrastructure? Though of course the reason that a variant is needed would be architecture specific.
More like 1...0, which nicely improves depth precision. Annoyingly due to symmetric -1...1 range reverse-Z cannot be used on OpenGL out of the box, but it can be fixed with ARB_clip_control. https://developer.nvidia.com/content/depth-precision-visuali...
Why should they get flak for having internal APIs? The fact that the internal API is a superset of the external API is smart engineering.
Think about it this way: Apple could just as well have made the "Metal that Apple uses themselves" some arcane "foocode" IR language or something, as I'm sure many shader compilers and OpenGL runtime implementations do, and nobody would be nearly as mad about it.
The fact that they use internal APIs for external apps in their weird iOS walled garden is obnoxious, but having private, undocumented APIs in a closed-source driver is not exactly an Apple anomaly.
It's not about having good segmentation of user-facing and kernel-side libraries, no one faults them for that.
It's about Apple building user-facing apps that use the whole API, and then demanding that other developers not use the features required to implement those apps because we're not trusted to maintain the look-and-feel, responsiveness, or battery life expectations of apps on the platform.
I hate a walled garden as much as the next developer, and the median HN reader is probably more than trustworthy. But past performance does predict future performance.
I am really thankful to the Asahi Linux team, and specifically in this instance for the GPU, [Alyssa Rosenweig](https://github.com/alyssarosenzweig), [Asahi Lina](https://github.com/asahilina), and [Doug all Johnson](https://github.com/dougallj).