I want to talk about WebGPU
cohost.org
cohost.org
First, for AI and machine learning type workloads, the infrastructure situation is a big mess right now unless you buy into the Nvidia / CUDA ecosystem. If you're a research, you pretty much have to, but increasingly people will just want to run models that have already been trained. Fairly soon, WebGPU will be an alternative that more or less Just Works, although I do expect things to be rough in the early days. There's also a performance gap, but I can see it closing.
Second, for compute shaders in general (potentially accelerating a large variety of tasks), the barrier to entry falls dramatically. That's especially true on web deployments, where running your own compute shader costs somewhere around 100 lines of code. But it becomes practical on native too, especially Rust where you can just pull in a wgpu dependency.
As for text being one of the missing pieces, I'm hoping Vello and supporting infrastructure will become one of the things people routinely reach for. That'll get you not just text but nice 2D vector graphics with fills, strokes, gradients, blend modes, and so on. It's not production-ready yet, but I'm excited about the roadmap.
[Note: very lightly adapted from a comment at cohost; one interesting response was by Tom Forsyth, suggesting I look into SYCL]
On NVIDIA GPUs, flops without tensor cores are ~1/10th flops with tensor cores, so this is a pretty big deal for inference and definitely for training.
Tensor cores: I can't say there are plans to add it, but it's certainly something I would like to see. You need subgroups in place first, and there's been quite a bit of discussion[1] on that as a likely extension post-1.0.
If you have a wishlist, we have an issue tracker! ;) https://github.com/gpuweb/gpuweb/issues
* "WebGPU powered machine learning in the browser with Apache TVM" - https://octoml.ai/blog/webgpu-powered-machine-learning-in-th...
* "Fastest DNN Execution Framework on Web Browser" https://mil-tokyo.github.io/webdnn/
* "Google builds WebGPU into Chrome to speed up rendering and AI tasks" https://siliconangle.com/2023/04/07/google-builds-webgpu-chr...
> Fairly soon, WebGPU will be an alternative...
So while the blog focused on the graphical utility of WebGPU, the underlying implementation of WebGPU is currently about the way that websites/apps can now interface with the GPU in a more direct/advantageous way to render graphics.
But what you're suggesting is that in the future new functionality will likely be added to take advantage of your GPU in other ways, such as training ML models and then using them via an inference engine all powered by your local GPU?
Is the reason you can't accomplish that today bc APIs haven't been created or opened up to allow such workloads? Are there not lower level APIs available/exposed today in WebGPU that would allow developers to begin the design of browser based ML frameworks/libraries?
Was it possible to interact with the GPU before WebGPU via Web Assembly?
Other than ML and graphics/games (and someone is probably going to mention crypto), are there any other potentially novel uses for WebGPU?
Yes.
> Is the reason you can't accomplish that today bc APIs haven't been created or opened up to allow such workloads? Are there not lower level APIs available/exposed today in WebGPU that would allow developers to begin the design of browser based ML frameworks/libraries?
That is correct, there is no way before WebGPU to access compute capability of GPU hardware through the Web. There have been some hacks based on WebGL, but those are seriously limited. The fragmentation of the existing API space is a major reason we haven't seen as much progress on this.
> Was it possible to interact with the GPU before WebGPU via Web Assembly?
Only in limited ways through WebGL - no access to workgroup shared memory, ability to do random access writes to storage buffers, etc.
> Other than ML and graphics/games (and someone is probably going to mention crypto), are there any other potentially novel uses for WebGPU?
Yes! There is research on doing parallel compilers on GPU (Aaron Hsu's co-dfns as well as Voetter's work[1]). There's quite a bit of work on implementing Fourier transforms at extremely high throughput. Obviously, physics simulations and other scientific workloads are a good fit. To me, it feels like things are wide open.
Only with tons of restrictions by going through WebGL, or by writing your own or extending an existing WASM runtime outside the browser which connects to system-native GPU APIs like D3D12, Vulkan or Metal.
> are there any other potentially novel uses for WebGPU
In general, the only thing that WebGPU has over calling into D3D12, Vulkan or Metal directly is that it provides a common subset of those 3 APIs (so you don't need to write 2 or 3 implementations to cover all popular operating systems) and it's easier to use (at least compared to D3D12 and Vulkan).
WebGPU is an important step into the right direction, but not a 'game changer' per se (at least outside the browser).
Have a look at wonnix https://github.com/webonnx/wonnx
A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web
I don't get it yet, but HN seems excited about it, so I'd like to understand it.
What I get so far - running models that can be run on consumer sized GPUs will become easier, because users won't need to download a desktop app to do so. This is limited for now by the lack of useful models that can be run on consumer GPUs, but we'll get smaller models in the future. And it'll be easier to make visualizations, games, and VR apps in the browser. Is that right and what other popular use cases where people currently have to resort to WebGL or building a desktop app will get easier that I'm missing?
If your app needs CUDA, you'd need to write it in CUDA.
If you don't need CUDA, you'd write it for WebGPU instead?
If you benefit from CUDA but don't strictly require it, then you can write it in CUDA with a WebGPU fallback.
Is that right?
Not only that, WebGPU is OS and hardware neutral. It'll work regardless of what machine you have. Currently it's Nvidia or get lost.
It points out that MoltenVK doesn't work very well, and I agree, although I think it's questionable whether the Molten-compatible subset is worse than WebGPU or not.
It says that Vulkan has driver issues on Linuxes, but provides no citations for that, and in my experiences, Vulkan drivers and developer tools have been the best for Linux, overall (my experience is limited to Nvidia, so maybe that's why, Idk).
WebGPU is basically a middleware for those that just want to use graphics, not all of unity, unreal, etc...
Yes, the Switch supports Vulkan alongside OpenGL 4.6, but if one wants the full power of the games console, the name of the game is NVN.
As for Apple, all relevant middleware has support for Metal.
Many companies have tried to unseat Nvidia. Many of them big companies. There is no magic in the ones you named. Worse, I'm not clear that Google will bring stability to the project. Such that I see little reason to be overly excited.
Happy to be proven wrong.
Coming at it from a graphics processing perspective, working on a lot of video editing, it's annoying that just as GPUs start to become affordable as people turn their back on cryptobro idiocy and stop chasing the Dunning-Krugerrand, they've started to get expensive again because people want hardware-accelerated Eliza chatbots.
Anyway your choices for GPU computing are OpenCL and CUDA.
If you write your project in CUDA, you'll wish you'd used OpenCL.
If you write your project in OpenCL, you'll wish you'd used CUDA.
Only if one is stuck in a C for GPGPUs mindset.
Stolen. Genius. You made my day.
Credit where it's due, though, it's an awesome term.
...tbh, I wish WebGPU would look even more like Metal, because the few parts that are not inpired by Metal kinda suck (for instance the baked BindGroup objects - which requires to know upfront what resource combinations will be needed at draw time, or otherwise create and discard BindGroup objects - which are actual 'heavy weight' Javascript objects - on the fly).
Also, this seems like some major revisionist history:
>This leads us to the other problem, the one Vulkan developed after the fact. The Apple problem. The theory on Vulkan was it would change the balance of power where Microsoft continually released a high-quality cutting-edge graphics API and OpenGL was the sloppy open-source catch up. Instead, the GPU vendors themselves would provide the API, and Vulkan would be the universal standard while DirectX would be reduced to a platform-specific oddity. But then Apple said no. Apple (who had already launched their own thing, Metal) announced not only would they never support Vulkan, they would not support OpenGL, anymore.
What I remember happening was that Apple was all-in on helping Khronos come up with what would eventually become Vulkan, but Khronos kept dragging their feet on getting something released. Apple finally got fed up and said, "We need something shipping and we need it now." So they just went off and did it themselves. Direct X 12 seemed like a similar response from Microsoft. It always seemed to me that Vulkan had nobody but themselves to blame for these other proprietary libraries being adopted.
This is not really how it happened. AMD released Mantle back in 2013 based off their experience with game console specific APIs. From what I remember, AMD expressed some interest in Mantle becoming a cross-vendor standard, but were a bit wishy-washy early on. GDC 2014 then saw some AMD talks on Mantle, the announcement of DirectX 12 from Microsoft and the AZDO talk. Apple then announced Metal in June of that year. The "Next Generation OpenGL Initiative" kicked off around that same time with a public call for participation in August. Apple did join the working group at some point (they're one of the many companies listed in a slide from the announcement presentation), but I don't see any evidence that they were ever a major player in the standard.
Now obviously, Vulkan was not an option for Apple in 2014 when they announced Metal since the project hadn't really gotten started yet, but I don't see any evidence that they pushed Khronos to get started on a replacement for OpenGL earlier either. They were also lagging behind on OpenGL support for years before they announced Metal (they stopped at 4.1 which was released in 2010) and notably almost none of the techniques presented in the AZDO talk worked on Mac OS for this reason. I'm sure part of the reason they decided to go their own way with Metal is that you can move faster as a single company, but I think it would be naive to assume that making cross-platform development between iOS and Android harder wasn't a factor.
And while Metal was released in 2014, Apple had already stopped updating OpenGL way back in 2010.
Apple is also supposedly engaged in a legal dispute with Khronos, which is why they so vehemently rejected SPIR-V in webgpu.
Mantle (which is what became Vulkan) also came out before Metal did (2013 vs. 2014)
> Direct X 12 seemed like a similar response from Microsoft
That seems like a stretch since nobody ever expected Microsoft to do anything aligned with Khronos. They'd been doing their own thing for a decade+, why would you think DX12 was anything different?
Is there any evidence for this?
Timeline? Metal first came out 2 years before Vulkan.
Seeing as Apple had already stopped updating OpenGL versions about 4 years before the release of Metal, it seems more likely that Apple never planned on working with Khronos on anything.
You also won't find any hint of involvement from Apple in anything Vulkan-related. Or really anything else Khronos-related. They've even pulled out of OpenCL - the thing that only Apple ever cared about in the first place.
No idea why they are still a member of the group. Possibly they just haven't been kicked out yet, possibly they still want to retain voting input for something (like webgl).
Sony cares about Khronos on Android phones, not so much on Playstation.
Nintendo supports OpenGL 4.6 and Vulkan on the Switch, with NVN as the main API. During the previous generations they started to support GL like APIs, not really 1:1 to the standards.
Microsoft was the main contributor to the initial set of GlTF 2.0 improvements, and BabylonJS is the first browser engine to driver most of the WebGPU efforts, and they care about Khronos APIs in the context of WSL.
Apple once cared about Quickdraw 3D, OpenGL was the path out of the tiny market they were in and what NeXTSTEP used anyway, nowadays all relevant game engines support Metal anyway.
Note that WebGPU in Chrome is not yet available for Linux.
Firefox Nightly has some partial support for Linux so that the first two of these examples work for me: https://webkit.org/demos/webgpu/
Try these instead: https://webgpu.github.io/webgpu-samples
How many games target SVG as their graphics pipeline?
How many applications, for that matter?
The comparison to PDF was better than you probably realize. PDF is Turing complete, and there have been ray-tracers implemented in it.
https://blogs.igalia.com/nzimmermann/posts/2021-10-29-layer-...
Please leave my drink alone. :-(
PostScript ray tracer:
Damn, and here I was going to send it to my LaserWriter IINT!
What ugly prejudices so many people on this thread seem to have. And how trivial and nonsensical. All I said was she should not have omitted svg in her history of graphics apis since she's writing an article about browser graphics. To deny that svg is a part of that, or that people don't program to it... well, I was going to write "ridiculous" but the xkcd about people who don't know things comes to mind...is it possible you just don't know that so many people make things with SVG? In any event, it seems I've touched a nerve that I didn't want to, and I find it ugly, and this is my free time, so I'm leaving the convo.
Peace.
You are continuing to double down on a straw man, which is why you’re getting all the push-back. The ugliness is ugliness you’ve single-handedly created because this is not an article about the history of browser graphics. It would have included SVG if that were the case, but it’s not, and you’re making assertions that aren’t warranted or justified here. This was an article about GPU APIs. Canvas was only mentioned briefly off-hand in passing twice, and the history of Canvas was not discussed at all, nor was the history of browser development. Nobody denied that SVG is part of anything, you are projecting your off-topic wishes onto a discussion where SVG simply does not belong, it’s tangential and not relevant to the article. It’s not relevant here how many people make things with SVG.
SVG is great. I wish development for SVG2 hadn’t stalled out, I would like to see SVG2 become broadly supported, there are some new options for dynamic scaling I’ve wanted to use for ten years. That said, this is a completely separate topic from the article & thread. You could get your SVG fix by submitting an article to HN that’s actually discussing SVG rather than trying to hijack the comment thread on an article about GPUs.
Well, yes, but you can see how your original comment may have been read as dismissive of other people's work?
Metal isn't a text based programming language like PostScript, and it has a completely different graphics model.
PostScript isn't pixel oriented, or 3d oriented, and Metal doesn't have a stencil/paint graphics model or text rendering.
They're totally different things, by totally different people and organizations.
Metal uses SPIR-V binary byte code, which is nothing like PostScript, a stack based high level polymorphic cross between Lisp or Forth, and a descendent of Interpres, JaM, and Design System.
As programming languages go, and as imaging models go, PostScript and Metal are at completely different ends of the spectrum.
SVG, like PDF, is the PostScript stencil/paint imaging mode, without the Turing complete programming language, but expressed as XML, with some CSS thrown in. And some implementation of SVG even let you define event handlers in JavaScript, but Metal sure doesn't let you do that. And to top it off, SVG is "object oriented" (i.e. a DOM) while Metal is "immediate mode" (i.e. an API). They couldn't be more different.
Since your facts are wrong, and you're throwing around insults like "ugly prejudices" and "trivial and nonsensical" and "ridiculous" and "people who don't know things" and "I've touched a nerve" and "I find it ugly", then it's a good thing that you're "leaving the convo", because you haven't contributed anything but misinformation and insults.
More about PostScript and its history:
It certainly doesn't, except that they're both LLVM based. As the article says, Apple can't use anything from Khronos.
Despite the name, WebGPU isn't limited to browsers, I think the article sort-of implies that without explicitly pointing it out (until late into the article).
Instead WebGPU has a good chance to become the 'missing link' standard cross-platform 3D API that sits above D3D12, Metal and Vulkan and at the same time is relatively straightforward to use for mere mortals.
If the article is going to bring in Canvas, it should bring in SVG. SVG shares more with opengl than Canvas does.
SVG is declarative.
Canvas and OpenGL/Vulkan are imperative.
svg's primitives are shapes and you program it with js and dom.
canvas primitive is a 2D array of pixels, and you program it with js and an api.
webgpu's primitive are pipelines (or whatever) and you program it with js and an api.
Maybe you're hung up on using the dom instead of an api. But the dom is an api, and it is further specialized by svg. Mutating the screen is exactly equivalent to mutating the dom within SVG. The mental model is that of a persisting scene that you nudge around with changes. This, versus the mental model of canvas where shape persistence is up to you. My understanding of low-level 3d graphics (limited to that of a hobbyist) is that driver commands setup a pipeline and then nudge around a scene graph in a similar fashion to svg. So it is doubly ironic that she would omit it.
"In the browser there are now 3 such tools: svg, canvas, and webgpu."
That's wildly incorrect, and anyone should take your comments with a huge grain of salt. There are far more than 3 drawing tools in the browser.
Why are you excluding CSS? You can draw with it, as well. https://codepen.io/matheuswf95/pen/wgRMwW
Why are you excluding Video HTML elements? It's possible to pipe directly to them from Javascript.
Why are you excluding drawing in the DOM? Do you hate ASCII and ANSI art? https://www.crummy.com/software/ansi2html/
Why are you excluding the JS Console? You can draw to that. https://dev.to/shriji/game-on-console-log-5cbk
Why are you excluding the HTML Title? Someone made a game that you play entirely in the browser's Title? https://titlerun.xyz/
Why are you excluding GIF? I can't find it, but I saw someone made a remote desktop client that encoded the video as a never-ending GIF stream.
Why are you excluding fonts? You can create custom fonts and use them to draw all kinds of crazy things.
We draw the line somewhere. I thought the History of Graphics APIs in the article was great. Sorry it disappointed you. I felt your attack on the author's credibility, "[I]t omitted SVG entirely. So one should take it with a grain of salt" was pedantic and rude.
Does that mean that HTML tables are a graphics API?
Maybe you do and that's fine. I program SVG by dynamically generating it on the server in Lisp. No js required unless I need to change an element at runtime. SVG is much more centered on "paint a picture with vectors and then send it to the browser" than "call a function to draw this vector RIGHT NOW" which is why people here are saying it's not really an API the way Vulkan and WebGL and WebGPU are.
For SVG to be considered an API, HTML would have to be one too. For SVG to be relevant to TFA, it would also need to express 3D geometries, and SVG is focused on 2D.
With WASM-focused initiatives to create hardware accelerated UI in the browser, we may soon see a toolchain that deploys to a WebGPU canvas and WASM in the browser, deploys native code linked to WGPU outside the browser, and gives the industry a roadmap to Electron-style app development without the Chromium overhead.
In some ways the problem with everyone trying to render to canvases and skip the DOM starts from education and a lack of native equivalents to the DOM that really genuinely showcase the strengths beyond having a similar API. I think developers come into the web and they have a particular mindset about graphics that pushes them away from "there should be a universal semantic layer that I talk to and also other things might talk to it", and instead the mindset is "I just want to put very specific pixels on the screen, and people shouldn't be using my application on weird screen configurations or with random extensions/customizations anyway."
And I vaguely think that's something that needs to be solved more by just educating developers. It'll be a problem until something happens and native platforms either get support for a universal semantic application layer that's accessible to the user and developers start seeing the benefits of that, or... I don't know. That's maybe a long conversation. But there has to be a shift, I don't think it's feasible to just hold off on GPU features. At some point native developers need to figure out why the DOM matters or we'll just keep on having this debate.
People wanting to write code that runs on both native and the web is good, it's a reasonable instinct. Electron-style app development isn't a bad goal. It's just how those apps get developed and what parts of the web get thrown out because developers aren't considering them to be important.
A lot of the "web frontend churn" meme is derived from the quest for a slightly better compromise - which has produced some results if what you are building resembles a web page. But over and over, native devs will say, "no, I want the hardware access. This layer still doesn't address my use case." That's been true of anything deeply I/O related on the Web - graphics, audio, input. Things where you can't make a simplifying assumption if you actually start covering all use cases.
I don't think this is true for the majority of native apps I run. I think a lot of developers think they need low-level control over exactly what pixels on the screen turn what colors, but I don't think most apps are in that category. Devs that use the GPU a lot are just used to working that way.
There are cases where that kind of access is needed. I use WebGL right now, I will use WebGPU when it comes out. But how much of your desktop software genuinely, legitimately could not be described using a pure-text tree? Maybe Blender? Even an app like Krita really only needs precise control over pixels for the actual drawing part.
It's mostly games and very experimental apps and visual content creation tools. Everything else is at most "we need a buffer rendered via the GPU embedded in this interface for one thing".
My take on this is: if your app works with a screenreader, that app is probably representable via some kind of universal interface. And the advantages to having a layer like that on native platforms would be profound. People talk about how great the terminal is, imagine if you could do everything you can do on the terminal -- piping output between programs, scraping content, building wrappers around interfaces, easy remote access, etc -- imagine if you could do all of that for all of the GUI apps on your computer too.
What we would lose in developer flexibility to define an initial interface we would make up for ten-fold with the additional flexibility developers and users would have to build around existing interfaces and to extend them.
> A lot of the "web frontend churn" meme is derived from the quest for a slightly better compromise
I'm also not sure this is entirely true. Yes, performance is a thing people talk about but systems like React came out of developers wanting a declarative model for building interfaces. In many ways it was going the opposite direction of canvas/webGL code. A lot of other libraries are similar -- you look at something like d3, that's not really about how things get drawn on the screen, what makes it special is data bindings.
Debates about the virtual DOM are kind of a small subset of the churn that we see online, and my impression is that the churn is pushed more by people debating how to build interfaces in the first place and how to further abstract that process, not how this stuff should get rendered to the screen.
We can have a high diversity of tools for building interfaces, what makes the DOM special is that it's universal render target. Whatever tool you use, the user gets the same semantics. And the DOM isn't necessarily a good universal render target. But it's kind of telling that even though the DOM has a ton of issues, the underlying philosophy of "you should have to describe what you want to put on the screen using text first and then we'll let you style it" is itself really empowering for end-users.
Because HTML+CSS+JS provides a fantastic cross-platform UI toolkit that everybody knows how to use.
Videogames create their own UI in order to have lots of shiny effects and a crazy immersive audio-filled controller-driven experience... but non-videogames don't need or want that.
Heck, I'm actually expecting the opposite -- for the entire OS interface to become based on Chromium HTML+CSS+JS, and eventually Electron apps don't bundle a runtime, because they're just apps. My Synology DSM is an entire operating system whose user interface runs in the browser and it just... makes sense.
Because UI is such a pain, it's actually not uncommon for browser-based games to use HTML/CSS to handle their menus and HUD elements.
My only critique in that area is that I can't put shaders on the DOM. CSS shaders were part of the CSS spec but they turned out to be pretty big privacy/security risks I guess, so they got deprecated before they ever really landed. I never figured out the full story.
There's post-processing that you can do on a canvas that you can't do on a DOM tree -- not because it wouldn't be possible to do, just because... for multiple reasons, some of them very good, the browser doesn't support that kind of thing. If you could do fragment/vector shaders on parts of the DOM, I would very likely be using it for basically every text element in my game's interface.
Those parts of the interface aren't really a performance concern either. The parts of my game where I feel like I really want GPU code are sprite rendering, the main gameplay window, etc... the menus are fine, I just don't like that I can't throw a fragment shader on top of them.
It's one thing to build panels and buttons for a video game. It's a totally different beast to build a text layout and editing and selection engine, accessibility and screen reader compatibility, clipboard support, date pickers and file dialogs, and ten thousand other things.
A lot of it is about text, I guess!
Once you've re-implemented all that in a canvas, you've more or less built another web browser, so why not just use HTML?
I agree that it would be kind of silly to then put this in the browser. Instead, I think it could be used to build the native target.
And just to clarify my earlier statement, I'm not a fan of phasing out the use of the DOM in the browser— it has graceful degradation and accessibility features that are really important. It's more that there are people with existing Qt or JavaFX or Cocoa dependent projects, or whatever, who are now able to deploy to the web by implementing the UI layers in WASM/WebGPU.
Let’s build a browser inside of virtual assembly code that runs in a browser that runs on an OS that runs on real assembly code (sort of). We could run that on a physical processor, or in a VM (hey, maybe we could run that VM inside a browser that runs on…)
I don't care about HTML and CSS and JS in specific, but yes, holy crud I want my native OS to be based on a user-inspectable XML-like tree. Yes please.
It doesn't have to be HTML if everyone hates HTML. But there is a lot of potential beyond just developer convenience to having native apps start representing their interfaces using semantic language. I don't know how to get across how much stuff that would enable on the computer. I need to sit down at some point and try to build proof-of-concepts because the idea is something I genuinely just randomly get excited about sometimes, even though I know it's probably not going to ever actually exist any time soon and nobody is working on it.
But just off the top of my head:
- I would like to be able to have extremely low-latency remote-desktop access by streaming only the "DOM" tree instead of the entire screen.
- I would like to be able to apply user-CSS styles to every single app on my computer. GTK/Gnome styling/etc... sort of works in this direction but... it's not as good. Let me right-click on a running app and inspect element. Let me style stuff in real time. This is something that was always a part of native development and still kind of is, but it feels like the native platforms have all gotten worse in this regard (but maybe that's just anecdotal).
- I would like to be able to pipe the interface of an app into another app and scrape its content.
- I would like to be able to script my native apps by running DOM queries rather than by learning some plugin language or simulating mouse movements.
- I would like to be able to stream a specific sub-tree of an app's interface to another device. I would like to be able to open up my phone and have it show specifically the color picker from Krita and have it pipe my input back into Krita and I don't want to learn a Krita-specific scripting language to do that, I just want to run a query and pull out that sub-tree and stream it. I would like to be able to have an app just pull out the "canvas" element from Krita and occasionally snapshot it or mirror it to another monitor.
- I would like to have multiple "views" of a single application by just passing the same application's "DOM" into two different renderers without needing to have two processes for the actual application.
- I would like to be able to do all of that on-the-fly without sitting down and writing some specialized program, the same way I can do stuff on a website by just right-clicking and going "inspect element."
And again, I don't really care if it's the DOM specifically, I just want all the stuff the DOM provides me as a user.
I mean, that's been the case for most GUI toolkits I know of, for decades before electron was a thing, and part of why I've never been impressed by the value proposition of DOM and web in general as a GUI toolkit/rich application platform by people with no prior exposure to proper GUI frameworks.
Qt has GammaRay JavaFX has Scenic View Gtk has a built in inspector (Ctrl+shift+I) Windows has things like UISpy/Inspect
Yes and no? Or at least, yes and to the best of my knowledge, no.
Yes, GUI toolkits do have similarities. But none of these tools have ever really been expanded out or utilized to the same degree that the DOM has been utilized. If you look at the documentation for most of the tools you link, they're primarily billed as debugging aids. They're also not something that the OS itself treats as an first-class feature.
I do think in particular GTK has been moving more in this direction, but my (possibly incorrect) take on it is that it's basically copying the web (which to be clear, is a good thing). GTK outright uses CSS now.
I'll also add onto this that I don't think the DOM is the equivalent of a GUI toolkit in the first place, I think it's a render target. It's at its best when it's targeted as a user-facing feature, not as a developer aid. That's not really ever been my perception of native toolkits (although to be fair, system theming has been a thing forever, but that's really just scratching the surface of what's possible here).
There's stuff I can do with a browser console and extensions that I can't do as a user with a compiled GTK app -- and that's even acknowledging that browsers themselves don't leverage the DOM to its full potential. But they're still miles ahead of what I can do with native apps as a user.
Not to say that these inspectors couldn't go in that direction. And I hope they do. Like I said, I'm not married to HTML in specific, I don't really care about how we end up heading in this direction, I'm mostly just pointing out that there's really serious advantages to having an XML-based text interface that's inspectable and that is separate from styling and those advantages get lost if everything just gets spit out as a pixel blob. Qt in particular is fun to bring up here because their web targeting makes the exact same mistake, so I don't think GammaRay is proof that they actually understand the strength of this model, because otherwise I don't think they'd be making this mistake when targeting the web.
What I will say is in praise of these tools is that the success of platforms like QT and GTK should on some level be evidence that "the web is for documents, not apps" is kind of nonsense. Most apps are documents. QT and GTK do just fine while forcing the developer to build a hierarchical UI of standardized components. They have no problem standardizing things like input handling. A pure-text tree-based UI with universal semantic components is something that works for most apps, even on native. I mean, GTK literally uses XML for interfaces. So I don't buy the whole "document/application" distinction that people sometimes pull out; everyone is already building interactive documents on native and it's completely fine and completely sufficient for a ton of serious applications.
I think you got the history backwards: the DOM was never designed to hold rich applications, controls and containers like a GUI toolkit does. HTML got forms (later, with HTML4), then organically grew more diverse and specialized controls (without ever reaching parity with most GUI toolkits in this area), never quite figuring-out containers nor behaviour inheritance (unlike in object oriented programming and the "Widget" model), while JS got steadily tamed and reshaped into a general-purpose language. I don't disagree with the assertion that "DOM has been utilized in extreme ways", I just don't think that it's a positive thing, simply because what we ended-up with is something extremely ad-hoc, inefficient and counter-productive (but as the saying goes, "when all you have is a hammer…"). What made the Web a successful platform is the "URL as deployment process", not that it's a great tech (neither on the producing nor consuming side).
> If you look at the documentation for most of the tools you link, they're primarily billed as debugging aids. They're also not something that the OS itself treats as an first-class feature.
Which is totally different than web browser's DOM appearing under "developer tools", or is it? Also, this ability to hook onto the DOM and play with things in the console disappears with packaged electron apps, which turn-out to be often less "introspectable" in practice than their native counterparts. Even on the "normal" web, this ability becomes a rarity as more and more pages get minified and bound to heavy SPA frameworks.
> my (possibly incorrect) take on it is that it's basically copying the web (which to be clear, is a good thing). GTK outright uses CSS now.
And it's not the only one, JavaFX and QML (Declarative Qt) can be built from XML and styled using CSS-like rules. Microsoft has had many attempts at declarative UI based on XML (XAML, UWP, …). XUL was Netscape solution to building applications with HTML at a time HTML wasn't ready (which lasted well into recent history, with 2018 IIRC being when Mozilla could finally go without it). Though let's not conflate everything: Web implies DOM, but CSS/XML doesn't, and defining documents and UIs in a SGML is much older than HTML itself.
To be clear, Electron is NOT what I am asking for. Electron is not fulfilling the spirit of having native DOM. I don't care what it's authored in, I care about what I can do with it as a user. Electron is in some cases a step backwards because I lose even the very limited styling options that the OS does provide.
It's not about what the developer writes, it's about what I as the user am presented with. The major strength of the DOM is that you have to describe your interface using text to the user, not to the compiler.
> Web implies DOM, but CSS/XML doesn't, and defining documents and UIs in a SGML is much older than HTML itself.
But this gets back to your original premise: the DOM was never designed to hold rich applications, and yet, it turns out that XML is perfectly fine for authoring rich applications. The whole document/application distinction hasn't held up (and arguably never held up), which is evidenced by basically every one of these toolkits pushing in that direction. I'm not sure how people can say that the DOM/CSS is insufficient for this when native toolkits are literally pulling CSS in now.
But again, I don't really care about HTML/CSS in specific. I care about the architecture. If the problem is just what specific components HTML provides, then use something else. But I want there to be something universal that (almost) everything on my OS uses. I want to get some broad agreement from application devs that they're not going to push pixels to the screen, they're going to give me as an end user a live interface in a shared XML format.
And I want to be really clear again, defining documents in UIs and SGML is not what I care about. I care about what gets presented to the user. The authorship is not the important part here.
I want to be able to open my calendar app's interface in a text editor.
You mention the browser's dev tools yourself, and it's a really good question that I think kind of demonstrates my point:
> Which is totally different than web browser's DOM appearing under "developer tools", or is it?
To be completely honest, yeah, I do think it's totally different. I can do things as a user in the browser dev tools that I can't do for any of these native toolkits: insert additional markup, add my own rules and event listeners and scripts, etc. And importantly, I can do that stuff on every single website, not using a specialized debugger for a small portion of the apps on my computer. I regularly open my dev tools and write little helper scripts for websites. I regularly alter CSS on the fly for websites I visit.
And sure, I don't actually think that browsers are doing a great job of leveraging this, I think they could go way further and could support even more powerful use cases. And some of the integration between the DOM and JS and the tendency towards div soup is actually quite bad on the web. But even so, the experience of working with the dev tools as an end user consuming somebody else's web app interface is miles ahead of the functionality on native platforms. In a lot of web apps that I use seriously I have extra CSS rules getting piped in through uBlock Origin. I don't have that on for any of my native Linux apps.
Yes it's still just called "developer tools", yes browsers could support more, but the entire philosophy around CSS as a user-extensible styling tool and HTML as something that can be queried and manipulated on the fly by multiple agents is something that's always been embraced more in browsers than on native platforms (limited OS styling options aside).
Same problem with WebView
- The Bevy game engine, using WebGPU on the backend. https://bevyengine.org/
- Wonnx, A WebGPU inference engine for running AI compute on the server or in the browser. https://github.com/webonnx/wonnx
http://kvark.github.io/web/gpu/native/2020/05/03/point-of-we...
It's actually better in an application than in the browser. In an application, you get to use real threads and utilize the computer's full resources, both CPUs and GPUs. In browsers, the Main Thread is special, you usually can't have threads at different priorities, there's much resource limiting, and the Javascript callback mindset gets in the way.
Here's video from my metaverse viewer, which uses WGPU.[1] This seems to be, much to my surprise, the most photo-realistic game-type 3D graphics application yet written in Rust.
The stack for that is Egui (for 2D menus), Rend3 (for a lightweight scene graph and memory management), WGPU (as discussed above), Winit (cross-platform window event management), and Vulkan. Runs on both Linux and Windows. Should work on MacOS, but hasn't been debugged there. Android has a browser-like thread model, so, although WGPU supports those targets, this program won't work on Android or WASM. All this is in Rust.
It's been a painful two year experience getting that stack to work. It suffers from the usual open source problem - everything is stuck at version 0.x, sort of working, and no longer exciting to work on. The APIs at each level are not stable, and so the versions of everything have to be carefully matched. When someone changes one level, the other levels have to adapt, which takes time. Here's a more detailed discussion of the problems.[2] The right stuff is there, but it does not Just Work yet. Which is why we're not seeing AAA game titles written in Rust. You can't bet a project with a deadline on this stack yet. As you can see from the video, it does do a nice job.
It's encouraging that WGPU is seen as a long-term solution, because it improves the odds of that work getting completed.
Still, I was chatting with one of the few people to ship a good 3D game (a yacht racing simulator) in Rust, and he admits that he simply used "good old DX11".
[1] https://video.hardlimit.com/w/tp9mLAQoHaFR32YAVKVDrz (That's on PeerTube. I'm curious to see how it copes with an HN-sized load. PeerTube is supposed to scale automatically.)
[2] https://www.reddit.com/r/rust_gamedev/comments/1302512/reall...
Does it actually work with OS features like screenreaders and printing that aren't just "pixels go on screen"? Does it break when a user has an OS setting changed that only one Microsoft QA greybeard remembers exists?
Protip: accessibility APIs are good for writing UI tests!
Anyway, congratulations to WebGPU release on stable Chrome on some of the platforms! Looking forward to see it widely available.
Lol, I wonder what their opinion is now 8 years later (at least those who haven't been burned out by Vulkan).
There's a reason so many cross-platform games of the era had some kind of demo or animation that'd run near the beginning; it was secretly the way to test card capabilities and figure out whether the hardware could do what it said it could.
It was interesting to read that Rust has a WebGPU implementation [0] outside of the browser. I wonder if this will become a more general standard.
Anyone know how WebGPU compares to CUDA in terms of performance and functionality?
Such a shame that they lobbied against SPIR-V on the web. Textual formats are evil.
Even without Apple in the way, SPIRV as it is wouldn't have been usable for WebGPU: http://kvark.github.io/spirv/2021/05/01/spirv-horrors.html
Yes, SPIR-V is weird and confusing... if you've never worked on a compiler. A lot of it is relatively standard if you have. Some of the messy control flow stuff got pretty massively cleaned up by the POPL paper. [0]
The issue is that Naga wants to support SPIR-V, but doesn't want to grow to become a full compiler. e.g.
> When writing SPIR-V, you can’t have two integer types of the same width.
This makes compilers much easier as now you can guarantee that all types are uniquely comparable by index. Just shove the type in an array, done.
> However, it can also declare an output “interface struct” with a bunch of special built-ins and an empty name… Perhaps, it’s a mechanism that helps with tessellation shaders. But for us in Naga land that caused more trouble than good.
No clue why you would care, unless you just blindly broadcast names when translating from SPIR-V. But also, don't do that? It's fine for things to have no names.
> Later I was pointed to the fact that all the existing production tools for generating SPIR-V (namely, glslang and dxc) only do so for a concrete entry point. So the multiple entry points is there in spirit, but discriminated in practice by the driver bugs, as well as this little validation rule.
Yeah, this is unfortunate. Multiple entry points are one of those things that make it seem easy to write a translator. Unfortunately, they're under-tested, not in the CTS, as glslang/DXC don't support them, and aren't how most shader pipelines tend to work.
Anyway, I don't want to enumerate all of this. SPIR-V has issues, sure, but it can be simultaneously true that SPIR-V isn't flawless, while also being true that it's the best version of this that anyone's seen yet.
[0] https://www.doc.ic.ac.uk/~afd/homepages/papers/pdfs/2023/POP...
I got really curious about this. To my understanding, I have been using Vulkan on my Linux desktop computers for quite some time now. What Linux devices could the author mean?
Did the name "XBox" come from the fact that in ran DirectX? Sort of short for DirectXBox?
MS called all their multimedia/gaming APIs DirectSomething for a while, then decided to group it all together into DirectX. It was also a time when you had to put an X into everything because you were XTREME.
Super Soakers, Jolt Cola, the Teenage Mutant Ninja Turtles, Sonic the Hedgehog, the FBI would come after you for reproducing copyrighted films (according to the warning at the start of the film). Definitely seems like an era in which something would be named DirectX.
Unsurprisingly, this didn't sell many Xboxes over there.
(Being racist is a continuing tradition with Western game journalists, and caused Japanese devs to genuinely think nobody liked them and try to Westernize their games for a generation or two. The director for the new Final Fantasy actually said he doesn't like it being called a "JRPG" because they think we use it as a slur.)
Also, once/if the WebGPU interface expands into more direct GPU functionality, I imagine there will many that will try to use your GPU for crypto mining/etc.
But surely a permission option/prompt will eventually be introduced, but probably not by google in chrome. Anyone know the roadmap/timeline for WebGPU on Firefox and other browsers?
No problem; just put such APIs behind a permission and they become unusable for fingerprinting.
I haven't played with WebGPU much (coughLinux supportcough) but I'm looking forward to it. And I generally agree that even more than the API itself, having GPU code that's easily portable between the web and native languages is a pretty big deal to me.
It's quite easy to almost lock up a computer by doing high intensity GPU work.
No matter what PC I use, I can't open that website...
Someone should probably tell Amazon that hyperlinks allow linking to other HTML documents, though, because that's a ridiculously long page.
I thought FF should not allow that to happen?
[ 7527.750745] HTML5 Parser[14186]: segfault at 0 ip 0000556be8b79cef sp 00007fbdb1570420 error 6 in firefox[556be8b77000+9c000]
I do not expect this to happen at all. WepGPU (including its shading language) is a subset of Vulkan. Furthermore, it is up to the runtime to expose vendor extensions to the code (as one example, Node supports ray tracing, but nothing else does). This means that WebGPU will be perpetually behind Vulkan.
That being said, if WebGPU does what you need then don't bother with Vulkan.
WebGPU does not support bindless resources making it a non starter as a Vulkan or D3D12 replacement.
(most current restrictions are enforced by the requirement to also work on mobile devices)
I fully agree with that, for a lot of use cases WebGL has everything you need, means it has the potential to become the cross platform graphics API OpenGL dreamed to be. And as a bonus you have a realistic way to run whatever app you are writing in the browser with WASM+WebGL.
I just think for AAA games Vulcan, Metal and DirectX12 will probably still be the way to go. But GUI libraries? Less highest end games? There is just no point once you can use WebGL everywhere. And then if you want to have a browser Demo you have a realistic chance to get it.
I think the original post refers to WebGPU
given that WebGL was known to use an "old" approach even before it was released and we don't have the problem with WebGPU I hope history doesn't repeats itself
All we need is the last mile -- progressive web apps -- to include better support for integration with desktop OSes and we have a way to take WASM apps and drag them to the desktop.
The up and coming languages for writing these WASM apps seem to be Go and Rust. Here's a Rust example:
https://github.com/gfx-rs/wgpu
https://dawn.googlesource.com/dawn
(these are the libraries that are used in Firefox and Chrome to provide the WebGPU implementation, don't know if Apple/WebKit also has open-sourced theirs)
Some non-browser WASM runtimes also have WebGPU support already, but unfortunately only for offscreen rendering or compute tasks (e.g. for writing cross-platform windowed apps as WASM blobs, there's a window system glue layer missing).
PBO support in WebGL lacks glMapBufferRange for non blocking read back.
In practice it'll only use Vulkan on Android and Linux, the main two implementations both default to DX12 on Windows even though they could technically use Vulkan there.
Why it needs cookies/XHR to display a page of plain text, I don't know, but I left the site.
--
[0] The aptly named "Javascript Toggle On and Off" plugin for Firefox sets up a keybinding for toggling. It's very convenient and great for scaling many a paywall.
I’d love to see a production architecture and file structure for this setup if anyone has a pointer to a GH repo or something similar
https://github.com/floooh/sokol/blob/4535a3b4be59eb912e77e04...
Web based graphics editor where the engine is written in rust. I think it uses tauri
What? Vulkan is supported on all relevant GPUs on Linux.
There is no "severe driver problem" either.
I stopped doing graphics programming somewhere between OpenCL and Vulkan being released and always wondered what had happened - especially along with the sentiments in the industry since that's difficult to glean from an wikipedia article.
Zawinski's Law: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."
Zawinski himself has stated:
"My point was not about copycats, it was about platformization. Apps that you "live in" all day have pressure to become everything and do everything. An app for editing text becomes an IDE, then an OS. An app for displaying hypertext documents becomes a mail reader, then an OS."
The dream is for Explorer/Finder/etc to no longer exist and for the whole computing experience to be something you just download and customize to your heart's content. Just imagine! A day when you can no longer tell you're using Windows unless you're unfortunately saddled with the job of making something run on it under the hood. That's the only way Microsoft's negligent idiocy is ever going to be shut down, anyway. I honestly can't wait.
(... although, hopefully in this new world browsers can run a language that isn't based on Javascript and applications can be built in a language that isn't based on HTML.)
Thank you. Somehow I had missed that expansion, but this makes the quote a lot more helpful.
Perhaps it should be called "Zawinksi's Trap" — to a programmer working on an app all day, the app becomes the operating system, leading them to justify expanding the feature set, which benefits them. For all other users where the app is not the operating system the app becomes more bloated and complex.
https://github.com/webgpu-native/webgpu-headers/blob/main/we...
The next missing piece is a standard window system glue API for WASI though.
VAO is the final GL feature.
There is no point to use anything other than OpenGL (ES) 3+ for eternity.
Copying the wheel over and over is the trap.
Vulkan is a Standard. It uses standardese language to define things. It defines all the relevant concepts like instance, physical device, logical device, command pools and buffers, etc. (Vulkan is not that different in this regard from other standards like C Language Standard, or C++ Language Standard).
The Standard also explains how this different concepts relate to each other, how GPU and Host interoperate, how memory flows, how synchronization works.
For example, it explains that there is a thing called instance, then this instance allows you to enumerate physical devices and obtain their properties, it explains that such devices have queues which can execute work. Then one or more physical devices can be used to create a logical device, which is a sort of lightweight context object. Then this logical device is used for allocating different resources, building pipelines, etc.
> Every Vulkan call involves passing in one or two huge structures which are themselves a forest of other huge structures, and every structure and sub-structure begins with a little protocol header explaining what it is and how big it is. Before you allocate memory you have to fill out a structure to get back a structure that tells you what structure you're supposed to structure your memory allocation request in....
Vulkan IS cumbersome to write using just C API, yes, but when you write actual program you usually wrap this functionality in a thin layer, and then suddenly everything becomes a lot less cumbersome. For example, you write initialization code exactly once, tailored for you specific needs, then wrap it in a function.
Construction of all of the pipelines, render passes, command buffers, etc. can also be wrapped in much the same way.
For example, construction of buffers and textures requires 3 steps: 1. creation of buffer/image object; 2. memory allocation; 3. binding memory to the object.
This functionality can be trivially wrapped in a function, then used many times. Then there are some libs like VulkanMemoryAllocator which simplify writing Vulkan even more. Memory synchronization is explained in https://github.com/KhronosGroup/Vulkan-Docs/wiki/Synchroniza...
The thing is, Vulkan requires planning. So if you want to write some simple game which displays sprites on the screen, then you are better off with something OpenGL-like. But once you need an optimized rendering pipeline with HiZ culling, minimal CPU-GPU memory transfers, etc., then Vulkan is the way to go. You can use newer OpenGL versions for this, but the code becomes much like something you would write using Vulkan API anyway.
FWIW i also disagree with the article on that front. I wrote a simple demo of Vulkan[0] the day the specs became available and was able to learn it (or enough of it to write the demo) just by reading the specs themselves - i remember finding them very easy to read. However i already had graphics programming knowledge using OpenGL for many years before that - but it isn't like someone is going to learn graphics programming from OpenGL's spec either, there is/was the Red Book for that.
That said, i agree that Vulkan itself is cumbersome to write - and really why i never bothered with it after writing the demo and getting my feet wet with it. IMO...
> Vulkan IS cumbersome to write using just C API, yes, but when you write actual program you usually wrap this functionality in a thin layer, and then suddenly everything becomes a lot less cumbersome. [..] Construction of all of the pipelines, render passes, command buffers, etc. can also be wrapped in much the same way.
...having to paper over an API to hide its guts doesn't exactly sound good for that API's design. Personally at least i never felt i had to paper over OpenGL, or even the little (pre-12) Direct3D code i wrote.
As a counter example, the "D3D Engineering Specs" do the same job, but are much more readable:
https://microsoft.github.io/DirectX-Specs/
PS:
> You can use newer OpenGL versions for this, but the code becomes much like something you would write using Vulkan API anyway.
Ironically, a lot of "modern" Vulkan might end up looking like old-school OpenGL again by using this new extension ;)
https://registry.khronos.org/vulkan/specs/1.3-extensions/man...
DirectX is not a standard so I don't think that makes this comparable. Most likely fewer people will have to read and implement it.
WebGPU is based on similar modern paradigms but it's an actually usable, sane, not overly verbose API. It's a pitty that WebGPU is deliberately gutted to make it a 5 year old smartphone graphics API because I'd prefer it over Vulkan any day.
I'm glad that Vulkan is taking some steps back on render passes, pipelines, etc. If they keep removing unnecessary entry barriers, I might eventually try the switch again in a few years.
It follows the tradition of Khronos APIs, having each developer writing that thin layer as rite of passage, one must earn their stripes to use Khronos APIs.