GPU-Backed User Interfaces
digest.browsertech.com
digest.browsertech.com
I’m exhausted by the modern web. There are so many layers and layers and layers, and none of it works very well!
But WebGL rendering on the web offers a unique and compelling alternative. I can write my frontend and backend in Rust. I can use gui libraries like egui directly if I want or write my own UIs if I prefer.
It’s almost like going back to the bad flash days, but with open web standards.
There are a bunch of downsides (some the article mentions. Accessibility isn’t there yet, the platform is immature so a lot of basic stuff you have to write yourself (form handling, etc), and a big one for me is: text rendering! There appears to be few if any mature text rendering libraries that you can drop into a webgl project and get good crisp text, shaping, and kerning.
But then I hit limits and started re-implementing the wheel. So I bit the bullet and learned all those layers of web development. Never looked back.
When you get to that point, may I suggest you master HTML and submitting forms. You can do a lot without Javascript. Look at Laravel for an easy to comprehend framework (PHP). It is just perfect.
Then for more complicated stuff I use self contained web components. That web component might be a terminal, or it might be a text editor, or it might be some kind of video calling widget.
One example, you don't need javascript for a chat interface (aside from htmx). You can use htmx to subscribe to a server-sent-event stream for receiving messages, and for posting them you just use a normal form element. It lets you write a lot more app-like functionality as standard html, and for when you can't do that a custom element is often the answer.
Don't know why web components haven't caught on more.
- figuring out when to redraw the screen or sections of it
- organizing your UI as a hierarchical tree that can be traversed
- handling events in general. what did the user click on? What Z order are things in? Can we find out using 2D boxes or do we need to do 3D ray intersection?
- combining UI hierarchies and events into event propagation
- rendering text and images crisply is surprisingly tricky
- blending layers over each other requires care.
- layout flows (margins, paddings, things pushing against other things) need managing.
...and a host of other things. For anyone interested, I wrote a summary in my early days of development here: https://medium.com/@WolframHempel_82303/seven-things-ive-lea... - though this misses a lot of what I figured out later.
It won't automatically render things as a tree (but the widget system does that for you) or doing layout flow (it's not a layout manager). But it does all the other things so well it's amazing it hasn't been cloned into every other graphics framework.
What happens if you just brute-force and re-render everything when there's a change? I'm all for optimizing but is this really required these days? Even considering laptops and mobile power requirements, rendering off-screen and flipping is lightning fast.
Layout flows are a different matter and can be expensive to compute.
I believe it is not speed, but battery life. Most of the time nothing happens on the screen besides some small widget changing like once per second (e.g. clock), doing 59 full rerenders every second for that (or even more with high refresh rate screens) is significantly higher drain.
But I am absolutely not an expert on this.
Does native OS frameworks have some escape hatch for these slightly changing, constantly damaging elements?
Of course it is. Even if it doesn't matter on your machine, your users will launch your code:
* on a 10 years old PC
* in train on a laptop with 10% battery left
* on a computer with so many things running in the background that you'll be on the edge of thrashing
* on a thermally throttled ultrabook
* on an underpowered SBC
* through foreign CPU arch emulation
* [insert more cases here]
In each of those scenarios, your code will either do its job well or be the source of frustration.
figuring out when to redraw the screen or sections of it
Info here: http://uggly.bytester.net/
I keep hearing that learning 2 or 3 native UI frameworks is too much work and too costly, and the answer keeps being to learn yet another new cross-platform UI framework that creates apps that are some combination of slow, lack useful features, and don't look right to the users. At this point, wouldn't it be less costly to just learn the native UI frameworks?
In GTK 4's case, drawing the widgets, CSS transitions, shadows, etc is all on the GPU. Video from GStreamer arrives as GL textures, same for WebKit integration and therefore can be used with all of the other high-level render-tree snapshotting features.
Indeed correct unicode text rendering is one of the difficult problems. It's the same thing that is missing from basically all games. So when people say stuff like games can run at 120fps, how hard can UI be? it's basically the same Windows 95 argument. The UI work being done is a small subset.
It really depends on the context.
If you look at it in a certain light, fonts are just "vector graphics" and there are ways to do that without using SDF (signed distance fields) and get really high quality precisely positioned fonts fully rendered on the GPU.
An example of that is the GLyphy project and we have a branch which can render fonts using that with GTK 4. It's a bit slower, but better quality and still easily able to handle 144hz full screen text scrolling. It requires some preprocessing before uploading glyph curves to a data atlas on the GPU.
For other shapes, there are options, but given GTK has so few upstream developers, they aren't quite ready for prime-time so they'll likely be part of GTK 5 in the form of "GskPath" if you feel like browsing the branches with that.
People seem to be really unwilling to do this, I don't know why. Maybe it's a language issue. Mind you, the official Microsoft story for how to write a native app is .. messy. But its XAML-based renderers do use the GPU.
Then you have both Google and Apple who invested massively into new languages and framework to facilitate the developments of apps and therefore native UIs
- Compose for Google in partnership with Jetbrains for Kotlin
- SwiftUI with Swift for Apple
Flutter in the way to make things cross-platform, all of them declarative, all of them native, all of them doesn't require learning a dedicated markup language
The problem are the politics between DevDiv (owns VS and .NET) with WinDev (owns COM, WinRT, C++).
WinRT came to be, as WinDev kept redoing Longhorn ideas in COM since Vista reboot, having .NET and C++ as the re-invention of .NET had it been fully based on COM instead of CLR (Ext-VOS original idea).
The new Windows 11 widgets started out as COM and C++/WinRT (which uses a primitive ATL like tooling), because only WinDev could come up with the idea of using COM, C++ and XML inline strings, for what Apple and Goole use nice Swift/Kotlin/Java APIs. Only after some backslash did they come up with the C# version of the API.
So we have WinDev doing WinUI 3.0/WinAppSDK, porting UWP out of the sanbox into regular Win32 stack, with lesser tooling (.NET Native, C++/CX deprecated, no designer), while DevDiv is focused on MAUI and Blazor (apparently putting into Web widgets is also a good idea).
First Microsoft needs to get rid of their internal GUI civil war.
They are usually _horrible_ to write in. Win32 is an archaic mess. Linux doesn't even _have_ a native UI framework, X and the like don't handle widgets. GTK is the least unpleasant but crafting UIs with it is very tedious. KDE is C++ based and interoperates poorly with anything that's not C++. Wrappers exist but they vary in quality. The Apple equivalent is almost pleasant to work with in comparison even though it has its warts (iOS you can deal with it directly in Objective-C no problem, OSX is more idiosyncratic)
And then you have to repeat for all supported platform and rewrite everything.
You could use something like wxWidgets, but it's not pleasant to use either. Language bindings are still problematic.
No? Basically all the GUI frameworks are horrible to write in, and if you haven't reached a point where you're pulling your hair out then your application just isn't complex enough. There is an unavoidable amount of complexity in writing GUIs and using a cross-platform framework means you get to wade through that unavoidable complexity N+1 times: once for each platform and then once for your framework!
Also desktop PWAs are IMO underrated and a really good way to distribute a desktop app without having to bundle a whole browser engine with your app. We do this for Construct (www.construct.net) - a complete game development IDE - and I think it's worked out great.
There are a couple of talks regarding CSS tricks to force GPU redrawing.
I remember seeing this kind of content on Google IO and WebDev talks.
Would need to also go treasure hunting to find them.
Here is an old article,
https://www.chromium.org/developers/design-documents/gpu-acc...
Browser makers are constantly updating their rendering engines and these pieces of wisdom do go out of date so it's important to verify them every few years.
What's the point of saying a UI is GPU-backed (which usually means a UI hand-crafted using mesh, texture atlas, shaders, minimal draw calls, etc.) if it's also grouped together with one of the most bloated and inefficient means to output pixels on the screen (HTML/CSS).
I love the web, not hating at all, but this is comparing apple and oranges.
BTW, the GPU-backed browser UI and GPU-backed OS UI, these come with a bunch of things that someone’s hacked UI doesn’t, namely transparent software fallback when a GPU isn’t available, full compatibility with the UI spec, accessibility support, and tolerance for a wide range of GPUs. People making their own UIs directly almost never do these things (I’m guilty as charged), and as such their UIs are often some combination of more brittle, device dependent, and limited to a narrower audience.
Have you read the article? They specifically mention how nowadays app are just a web page instead of being a GPU-Backed UI.
Trying to push the narrative that HTML / CSS is somehow a GPU-Backed UI is almost comical.
The browser is GPU-accelerated, it is not what anyone in the industry would consider a GPU-Backed UI.
> The browser is GPU-accelerated, it is not what anyone in the industry would consider a GPU-Backed UI.
Now this is a different argument than what you said above. I agree that browsers are what make it GPU-backed, not the HTML/CSS spec per se. But HTML/CSS is becoming effectively a GPU-backed UI in practice, by virtue of the fact that all the browsers use GPUs to render HTML & CSS. This is also true of the UIs provided by Windows and MacOS and Linux - the UI isn’t required to use the GPU, but the OSes will use GPUs for performance when possible. This is all anyone here actually means when they call HTML & CSS a GPU-backed UI.
Is there a meaningful difference once there doesn’t exist a modern browser that doesn’t use a GPU to significantly render web apps?
What I reject is the notion that “GPU-backed” is limited to more than what the words mean literally. If the implementation is using a GPU to render the UI elements, then it’s GPU backed. It doesn’t need to be hand-coded, or a one-off, or using shaders, or guaranteed by the UI spec.
This is exactly what I'm arguing against. We name stuff to represent something. The article very explicitly mention "GPU-Backed UI" as one of the alternative to app that are made with web tech (HTML / CSS). Because it is talking about a very specific way to build UI. And the name they choose is "GPU-Backed UI", which is what anyone would use too.
But you are arguing that a Web Page / Browser / HTML / etc is "GPU Backed UI" and should be group together. But then we have the issue of how we name that type of UI that they are talking in the article which is very obviously something different than what a UI built using a browser would be.
This thread is now hung on a gate-keeping technicality, and has lost the actual point, which is that, regardless of what you want to call it, html & css rendering have been getting faster due to the increasing use of GPUs, which is narrowing the gap between bespoke UIs and interfaces made with more standard toolkits.
Not saying I disagree HTML can be fast, in fact I think it's quite a bit faster than the vast majority of homecooked solutions. But tautological categorization isn't very helpful.
In any case, HTML surely meets the criteria. A much more significant gain is to be had via performance engineering JS than translating it as-stands to rust, wasm, or any other "closer to the GPU" thing. And if comparing two post-optimizations, the HTML/JS will be much easier to debug, style, and make accessible, with practically identical performance.
In fact the flexibility of JS/TS may make the process of discovering a higher performing algorithm/data structure easier, resulting in a faster implementation than had the same amount of effort been dedicated to a worse algorithm in a "faster" language.
Important to note JS can get pretty close to the metal itself with glsl where it matters, while keeping the rest of the app traditional web tech.
To me it rather seems like there are different definitions of what GPU-backed means.
>The browser is GPU-accelerated, it is not what anyone in the industry would consider a GPU-Backed UI.
In which industry?
As a former web dev I would consider any calculation GPU-backed if it is utilizing a GPU.
With that definition and considering html/css is used to build user interfaces, rendering in modern browsers could certainly be called GPU-Backed UI.
If this is a term that already has a different meaning to just the words it is composed of, that's fair to mention. But it doesn't help a discussion to escalate using wording such as "Trying to push the narrative ... is almost comical"
Now people argue that web page are "GPU-Backed UI".
I get the word might be misleading, but the article is talking about a specific way of building UI using the GPU, which does not include web page, so trying to fit the web into that category makes no sense.
In some loose sense every UI is "GPU-Backed", so how do we name that specific way of building UI that the article is mentioning? They choose "GPU-Backed UI" which if anyone would have told me those words I would immediately know what they are talking about and I would've never included HTML/CSS in that category (even if in some broad loose sense of the words it does fit).
Where to draw the line?
As long as the UI or it's underlying engine was built with GPU in mind and not as an after-thought to accelerate it, it can be safe to say it probably fit the "GPU-Backed UI" definition.
If you can easily attach a Shader (https://github.com/emilk/egui/blob/master/examples/custom_3d...), it is also a dead giveaway that it is a "GPU-Backed UI".
WebGL is different from HTML / CSS. You could have a "GPU-Backed UI" running in WebGL.
The fact that Electron apps are GPU backed is indisputable. Whether they are good GPU-Backed UIs is an entirely different question.
Which is very different than saying that it doesn't use the GPU. Virtually every UI solution is "GPU-Backed", but when I hear the term "GPU-Backed UI", I know exactly what type of UI architecture they are talking about.
Not necessarily. I've seen far too many cases where GPU acceleration actually made things "slower." Throughput != latency
Mind sharing what you used for backend / frontend of your desktop PWA? I'm planning to work on a similarly complex UI and have been searching around for best-in-class (or best-trade-offs) technologies today
Any books, tips, tutorials, whatever you can share that inspired your architecture decisions would be ___immensely___ appreciated.
Apart from a few extra services on the side, the main app is all static resources so no real backend to speak of. Again, engineered like a traditional desktop app with everything running locally, which also means the UI always responds promptly (no waiting for network) and it all works offline.
Maybe I should write up a blog post about how it all works, but not sure how many people out there want to hear more!
At some point with your own renderer you have to implement a large part of the DOM, styling and rendering layers. And why should a text editor team have to do that?
I think a lot of teams would be better off by learning how to make faster web apps, where there's a subset of HTML/CSS that approaches a tree of GPU textures with layout and style on top. Do things like: use fewer elements, very simple CSS selectors, `contain: content`, `contain: strict`, or `content-visibility: auto` everywhere possible, use shadow roots, use a fast vdom-less template system, and a framework where each component is its own render root, use virtualization for large lists...
If you do that you can also ship a fast web app. And as VS Code shows by popping up in more and more sites, being an actual web app has some very nice distribution and reuse advantages.
While I mostly agree that creators of today's web forget that there are a number of simple techniques to render fast pages, there are also a bunch of places where the HTML+CSS+JS web technologies serve us poorly.
* Content Editable is a terrible editing canvas
* Dom and CSS are pretty terrible core animation primitives
* DOM and CSS make for pretty poor interactive graphics capabilities
And in these cases WebGL offers some interesting alternatives to explore.
A fully GPU rendered web app is almost the worst of both worlds. You'll generally have massive code bundles and terrible a11y.
But why do people do this? I note that a big problem with the web, as-is today, is how anemic and inflexible HTML’s standard <input> elements are, which forces devs to build their own widgets (often poorly), maybe use a framework like Angular, and then eventually go all-in on web abstractions like Flutter/Blazor/etc. If HTML were better then many (but far from all) of these cases could be avoided - for instance, I don’t understand why HTML still doesn’t have absolutely basic widgets like a combobox, a single-line textbox that wraps, a date-range input, hierarchical drop-downs (optgroup is single-level), defined styling and non-text content of <select>, and so on… (though we did recently get color and range/sliders - though there’s still cross-browser quirks which hamper adoption).
I hated the days of ActiveX like everyone, but when you’re building an internal-only (I.e. IE-only) webpage it was neat that we could pull-in any of the standard Windows controls - and plenty others - into a page and it would just-work. And you got accessibility for free too, which is more than what you get with <canvas>-based frameworks still - or a multi-megabyte JS/WASM bundle download either (or an npm-based build system that breaks every time you update your dependencies…) ]. Okay yes I’m jaded.
On Linux, VS Code can't even resize without glitching and showing unrendered portions of the window like it's 1999. Nor can it kinetic scroll. Those are two expected highlights of quality GPU-backed software.
Hopefully running in an actual browser would make this better simply because you can choose a browser that isn't Electron.
They're the classic example of jumping from unoptimized HTML/CSS to GPU to try to fix performance as they moved the terminal to be GPU-based, and it doesn't seemed to have helped because that introduced rendering bugs that would have been impossible with HTML.
I only use VS Code as an example of how many places you app can be used if it's web-based.
That matters very little. By that logic, everything is GPU backed these days and rendered by a compositor, on all platforms.
> At some point with your own renderer you have to implement a large part of the DOM
No, not really. Unless you are saying "DOM" to mean any tree structure. And you might not even need that for your app, depending on its layout.
When people say "GPU rendering" in this context, think user interfaces found in games. They are directly rendered by the game code without any intermediate layers. And very often, without anything that one would even remotely recognize as a DOM (3D games have a scene graph, but that's still different).
> And why should a text editor team have to do that?
Ask Sublime Text and iTerm. They already do that to render their content (and implement native OS dialog controls for the rest).
> I think a lot of teams would be better off by learning how to make faster web apps
True, just because of the amount of effort. _Users_, on the other hand...
Most "native" apps, are native in the sense that they use some some UI toolkit that renders using the platform specific and typically hardware accelerated ui infrastructure.
With webgl and wasm, those same toolkits can now work in a browser.
As for accessibility, that is indeed a challenge. You would need to use some ui toolkit that supports accessibility features. Those of course do exist, browsers don't have a monopoly on accessibility.
A related challenge is the integrating that with the browser properly so that the native accessibility facilities in your OS can do whatever it needs to do (screen reading, enhancing contrast, etc.). This is not a solved problem. But it sounds like it could be a solvable problem that would require some changes to wasm and webgl.
Once you generalize the definition of browser, both ends of the spectrum coexist: from basic HTML pages to fully WebGL rendered WebAssembly apps.
WASM is getting better with every browser release. Likewise for webgl. It was perfectly fine for doing 3d games many years ago. It's just that we are stuck doing the same react/js/whatever crap that we've been using for the last decade or so. It's just awful how low the standards are for web UI. But it was never good enough for game consoles or mobile.
At this point, it's probably easier to run a browser as a wasm program inside of a modern browser just to render some legacy web app from way back than it is to get that app working with modern browsers. Why even bother? Just run IE 6 in an emulated browser on an emulated windows 95. And probably it would be fine in terms of performance too.
Javascript + css has always been a bit of a cringe fest in terms of capabilities. If you develop a UI that matters for a gaming console, a desktop OS, or anything else, you wouldn't use any of that. It's why native development is a thing for IOS and Android. The phones are fast enough to do web development. It's just that the whole space is too competitive for that to be even worth considering.
So, maybe it's becomes a better browser without that?
To widely varying degrees. Some do a lot of work on the GPU, others use only the rasterization features, some only accelerate the compositing. This unfortunately means that using simple surface level terminology like GPU accelerated doesn't sufficiently describe what is actually happening so that we could make meaningful comparisons between toolkits.
Ultimately I think what's needed is proper benchmarking of performance and measuring of battery life. A toolkit that uses GPU acceleration for a simple step might actually be slower than a full CPU implementation, because there is a cost to syncing your data between CPU and GPU. Also if the whole UI is rather simple, then even a solution that is built completely on the GPU might be slower than CPU, again because of the associated costs. If it takes 2ms to get everything set up on the GPU for a frame, then it doesn't matter if the GPU renders it in only 1ms vs 2ms on the CPU, as total time is now 3ms with the GPU vs 2ms with CPU only. It's a contrived example, but one that could easily happen without a focus on measuring.
Tauri + Vue/Svelte/Solid/Etc seems like a great way to build a decent desktop and web app, but you just don't have teams picking anything but React React React React.
All this is not suitable for web, but for thing like Elector - hell yeah. HTML to bitmap extension would be really welcome. Then you could render UI in WebGL or WebGPU. Adobe Flash had something similar if I recall correctly.
It may do that but generally it still uses Cairo.
BTW, I would not rely long run on OpenGL existence on Windows and Mac (especially).
In any case GTK working on Windows and Mac is an artefact rather than something really usable.
I suspect that efforts to support such multiplatform feature makes Linux native GUI development less developed.
Generally Linux cannot be considered as a desktop OS as it has no stable and uniform system UI (window manager, graphics, etc.)
Uh, no it doesn't. I literally wrote the GL renderer, I know what I'm talking about.
> In any case GTK working on Windows and Mac is an artefact rather than something really usable.
Maybe pre-4, which is why it got redesigned.
> I suspect that efforts to support such multiplatform feature makes Linux native GUI development less developed.
What?
> Generally Linux cannot be considered as a desktop OS as it has no stable and uniform system UI (window manager, graphics, etc.)
Huh? You're out of your mind.
Do we have such basic thing as `MoveWindow(wnd,x,y,w,h)` on GTK?
Seems like still not:
- https://stackoverflow.com/questions/58103333/set-frame-posit...
- https://stackoverflow.com/questions/65508569/how-to-move-my-...
This is literally the dumbest argument ever. It's one extra line of code. If you have the GDK X11 backend, get the XID and call MoveWindow() yourself. If you have a macOS Window, call [NSWindow setFrame]. I don't see why you expect GTK to be your abstraction for insecure UI practices.
The compositor should be in control of this policy, not applications. Which is why that API isn't there any more.
It means that GTK is not a desktop API.
X11 is not a desktop API either as you cannot build with it GUI consistent with other applications on target platform.
Linux is not a desktop OS. By any means. Dot.
Until we have linux_move_window() function.
To make an OS you need a system of desktop-providing tools. Windows is an operating system that provides a desktop stack to the NT kernel. MacOS is an operating system built on Darwin and Mach. Following this line of logic, there are dozens of opinionated Linux distributions that qualify for your definition of desktop OS. (RHEL, Fedora, Ubuntu, KDE Neon, et al.)
Linux systems are no different than Darwin or NT ones if you compare like systems. If you spend all day comparing kernel features to OS features, I don't think you're going to make any meaningful discoveries.
To be short:
First Linux company that will decide to provide stable API similar to `WndProc(window, params)` (Windows) or `class_addMethod(window,@selector)` (Mac/iOS) will win Linux Desktop war - will make the Linux GUI for years to come.
If someone knows such company - please let us know, at least I am willing to participate.
If we're talking about kernels with windowing capabilities, there are none. If you're asking about OSes with a reliable, documented graphics stack, RHEL is literally what you're looking for. It's the shitty, "Windows-ified" model of desktop development, and almost everyone ignores it for desktop use. You might argue that it doesn't qualify, but you can't argue that nobody has tried it before.
Users and UI developers don't care. It is part of OS and it is always there.
The only things I know for sure are:
1. Desktop OS is not a distribution but set of popular GUI applications that work on that OS natively including tested specifically on it.
2. Lack of stable window and graphics API leads to fractured GUI developers community. Keeping in mind that number of Linux GUI developers is in magnitude of times less than for Windows/MacOS, makes Linux GUI perspectives very grim.
In general "desktop Linux" is not fair to its most loyal users because of these. Users are forced to pay extra price for hardware just to be able to run those Electron applications - no real native options for similar functionality.
That's what a Linux distribution is.
> Keeping in mind that number of Linux GUI developers is in magnitude of times less
It's roughly proportional to the number of people using Windows or MacOS for serious networking applications, yes.
> Users are forced to pay extra price for hardware just to be able to run those Electron applications
What are you even talking about anymore? Linux is not responsible for shitty Electron apps, Windows and MacOS is. People wanted a univeral runtime that worked across both operating systems - neither Microsoft nor Apple budged. So the browser it is - it just came with the side-effect that other browser-enabled platforms can also run Electron. Linux is not forcing you to use bad software because Apple and Microsoft can't work together to fix desktop computing.
If you don't like the Linux desktop, fine. Go use 9front or something. It still exists though, you cannot deny it's usability because you don't like the way it does something. I dislike the way MacOS handles it's desktop, but that doesn't mean it's not a desktop OS.
https://discourse.gnome.org/t/how-to-center-gtkwindows-in-gt...
As I’ve said above, not all windowing systems have a concept of
global coordinates or allow direct positioning of the windowing from
an application. Wayland, for instance, does not.
There's a lot to complain about with Wayland, but this is one of the few changes that seems rooted in logic rather than laziness.
You then use the "color shader" to copy from the texture atlas into the destination FBO to both create the shape and color at the same time. Since you often do this for a whole text editor, you can copy all of the glyphs and color them in a single glDrawArrays()s.
When GLyphy lands, it's a bit different. You preprocess the bezier curves into a simplified format (which doesn't have overlaps) and upload those to a texture atlas (but for general data). Then use GLyphy's shader program to render them, similarly to how the color shader was used.
For more general paths, which is much slower but also still very neat, you would do something akin to Pathfinder.
Meanwhile QT continues to work great and they've largely solved the licensing issues that were a problem in the past. It's also a comparatively stable project and supports targets like android, iOS, rendering directly on to buffers (you can embed it into games). If you're starting a new project I think QT should be your default unless you have a good reason to use GTK.
I'm not qualified to assess the code quality of either project, but I have been impressed by QT for the few things I've needed to build a gui for. Things just work but if you need really specific functionality (like rendering directly to a framebuffer with no display server on embedded linux) it's not too hard to make it work. I'm honestly a bit mystified as to why anyone would choose GTK for a new project, unless they were specifically targeting the Gnome desktop and only the Gnome desktop.
Sort of?
Mostly just design patterns that are explicit to GNOME. And that is primarily a response to people being (rightfully or not) unhappy that too much GNOME was ending up in GTK itself.
What does this mean? `a11y` isn't some standard for making your app accessible, in fact it mostly seems to be guidelines for how to make your website more accessible. Is there an `a11y` native app checklist somewhere that Gnome is compliant with (and presumably QT/kde isn't?). Am I missing some protocol Gnome is implementing that QT/kde isn't?
Legitimate question, not snark, I've looked into in the past and I really don't see what Gnome's doing to comply with this that everyone else isn't.
> I really don't see what Gnome's doing to comply with this that everyone else isn't.
That's because Sun and GNOME created those layers two decades ago for Xorg.
And on Mac OS Cocoa use GPU acceleration as well.
As if you want to show something nice and fency you will need GPU acceleration. But still you can fall back to CPU rendering. There is Open GL on CPU in MESA. same for DirectX. It is shower for some fency stuff but good enought for basic UI
This is a great article! Writing a custom UI framework to render on the GPU is a big cost--but it has been worth it for us. It has let us build a UI-heavy terminal while continuing to match the performance of some of the fastest terminals on the market (such as Alacritty). There is certainly an initial velocity hit for new engineers as they have to learn the semantics of our framework, but overall I wouldn't say it has a big detriment on our velocity as a consumer of the framework.
The callout to a11y is a good one--we have basic a11y support but it is by no means robust. There are additional downsides to building your own GPU-rendered UI framework other than a11y that I want to call out: you have to build most of the interactions with the platform yourself instead of getting it for free when rendering using system APIs.
One example of this is properly supporting i18n. A robust framework should support RTL text, positioning IME / the emoji picker, so forth. While implementing these yourself is doable (we have decent support for IME at Warp, for example) it requires an engineer or two to spend a few weeks to implement.
Text rendering is also a beast, see https://www.warp.dev/blog/adventures-text-rendering-kerning-....
For Warp, this has definitely been worth it, but it hints that the community needs better cross-platform GPU-rendered UI frameworks so people don't need to consistently build them themselves.
On the one hand I think you could argue that if you have a lot of text maybe html would suit your needs better. On the other hand a lot of UIs just need some text and it’s so east for it to look like junk.
Ideally canvas with WebGL and WebGPU would sit at the bottom of the browser API layer cake, the DOM at the top and squeezed inbetween public APIs for text, shape, CSS+HTML rendering. But instead we got this weird canvas thing dangling off as another high-level DOM element.
I went to your GitHub (https://github.com/warpdotdev/Warp) in order to see if I couldn't figure out how to build it for Linux, but seems that's like a "issues" repository and I cannot find the source code anywhere, nor is it linked on the main website (https://warp.dev).
I'm guessing at this point that it's on purpose? The source won't be available and it's only a closed source product?
Indeed, "at this point" they've demonstrated that cross-platform development is a low priority for them. Even if they do ship a Linux client, I feel like it will be a hard-sell when there are so many other terminals treating Linux like a first-class target.
It’s interesting that you mention this now - I’ve been a Warp user for a few months but just yesterday started seeking out another option because the UI has been feeling sluggish in Warp.
I installed alacritty and kitty, and they both feel insanely fast, comparatively. iTerm is approximately the same as Warp.
I think the difference is significantly smaller than most people would intentionally target with UI paints. But when I press enter, it feels like warp just has a minuscule delay, where kitty has instantaneous printed the results within the same frame that my finger hit enter.
Do you know if there’s a specific # of ms that Warp is targeting for a UI response from user input? I’m not at all an expert in this domain, but am curious if it’s “all in my head”, or if there really is a difference here. Do you have similar stats for other terminals?
Edit:
Only asking because I know it’s hard to get the right balance between extra feature & speed. The autocomplete & syntax highlighting are really awesome features that I love about Warp, and those likely take many more CPU cycles. Warp is far more powerful than kitty in that way. But how many ms does it cost to have those features?
I know the answer isn’t the same for everyone about what the right trade-offs are between features and speed, and I haven’t made up my mind yet which is more important for me, to be honest. Warp is crazy impressive and has been a step up in functionality. Again, really love the work you’re doing!
In fact I probably reported and had ATI and nVidia fix bugs that even made it possible to have pixel aligned textures, partial refreshes of GL surfaces etc. It was all basic stuff, but it was always overlooked because "gaming"...
I find it hilarious that the GL UIs are all trendy now, I've done many of them professionally for the last 20+ years. One of first one was Sibelius v2, a desktop software for writing (engraving) musical scores, I converted the windows GD code to a GL backend all the way back then, rendering in textures etc etc.
Having said that, I'm working on an alternative GPU accelerated and retained-mode GUI called eepp (https://github.com/SpartanJ/eepp/) while I work on a new code editor similar to Zed called ecode (https://github.com/SpartanJ/ecode). The framework still needs a lot of work but presents some ideas that might be interesting to some. Retained-mode GUIs are a complex beast and it's hard to achieve simplicity (I think immediate-mode GUIs can't be beaten in that regard, but they have their own downsides). So I invite anyone interested in GPU accelerated GUIs to take a look at my projects (online demos are available), I'm looking for collaborators! (sadly C++ isn't very appealing these days, but my code-base is huge and I can't write everything from scratch in a new fancy and cool language).
Flutter draws to a canvas at 60 FPS and uses the GPU.
https://buildflutter.com/how-flutter-works/
Anecdotally, I see it getting faster and faster on new engine versions without app changes.
I don’t think they are going to have troubles performance wise for long.
This is actually an interesting read I came across by someone from their team recently which poses some hypothetical directions they could take things in the future depending on how the web platform evolves over time.
https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC...
In the past, Scaleform was close to ubiquitous, at least for large games. Scaleform is Flash-based, so it was a vector graphics engine with a JavaScript-like language. It was presumably faster and less bloated than a browser engine would be, as a game middleware from the Xbox 360 era. But from a developer experience point of view, it closer to a browser engine than to "[making] a usable UI out of triangles."
Scaleform is still pretty popular, but it has become popular to create game UIs with web technologies. You can find libCEF.dll in pretty much every game nowadays. Sometimes it's just used to render the update news or in-game store, but often the entire UI is HTML.
Just use Sciter, by default for rendering it uses Direct2d/DirectX (+ Vulkan||OpenGL) on Windows, Metal||OpenGL on Mac and Vulkan||OpenGL on Linux. With support of a11y by the way.
That depends on what you mean by "system UI".
Most system utilities on Windows are still using GDI. GDI is not GPU accelerated (other than BitBlt).
Same apply to MacOS - CoreGraphics there is pure CPU rasterization library.
And the same situation on Linux/GTK, system uses Cairo - pure CPU rasterization library.
Usually, on modern OSes, window/desktop manager is the only GPU accelerated thing - it composes window bitmaps on screen.
It is just that instead of XAML it uses HTML/CSS, JS/C/C++ instead of C# and works on all desktop platforms.
For the note: historically Sciter appeared before WPF and WinUI/UWP : https://sciter.com/sciter/sciter-vs-wpf/
By any means WPF and WinUI are not UIs provided by the OS. These are userland UI libraries. Same thing as Sciter.
Without WinRT there isn't WinUI, while WPF really only depends on DirectX 9.
[1]: https://jeremiahmorrill.wordpress.com/2011/02/14/a-critical-...
GTK 4 renders by default using your GPU. The only part that is CPU rasterized is if we don't have an accelerated path for it (rare) and glyphs are first rendered on CPU, uploaded to a texture atlas, and then GPU colored/blit from there on.
Most native UI projects on desktop OSes are not OSS.
He can't cook, buying cooking utensils, ingredients and fire, is just plain incompatible with making meals.
He...
Thats what every game does - draws a UI directly on the GPU. Game engines like Unity have in-built frameworks for creating UI and then can package the same application for Windows, Linux, playstation, etc.
Many people have (ab)used it to create cross-platform software. It's very high-performance, you just get large applications and load times.
Only briefly, but seems covered to me.
Edit: now I get it — they mention games, but not game engines as a means of creating other applications.
(author here) Part of the appeal of posting here is that I learn stuff like this :) Are there some good examples of these that come to mind?
That is oft repeated but having lived through those times that seems just untrue, at least on Windows which was far more dominant than it is today. So many applications, big and small, were using completely custom UIs instead of native things. From WinAmp to Microsoft Office to all of Adobe/Macromedia suites to all cross-platform apps using GTK/Qt (or god forbid, Java). One particular example is the pre-electron Spotify app, which was wonderful small single-exe thing.
But because conversion rates are typically low and because of how the math works out for a funnel product, there's a larger business impact depending on which 2.3% is cut out. (Ignoring the negative business impact of being non inclusive/a11y)
12% vs. 10% is a 20% decrease in performance.
100% vs. 98% is a 2% decrease in performance.
I'm not arguing against a11y. I'm just saying, that's why ADA exists. It's not in the business interest without the law. Also it's just good ethics to be accessible.
I'm not sure exactly what this 2.3% is (for "blindness"?), but significantly more than 2.3% of the population wear perscription glasses. Those people have visual impairment and use assistive technologies
Screen readers are not the only assistive technology. Especially on phones, many people use assistive tech to increase font weight or size. Ignoring these options will remove customers.
It is really difficult to do this correctly. The simplest systems to build with are the ones where you have a large visualization window and you only need some knobs to control it.
The things that you think are going to be tricky (say, skeuomorphic knobs) are actually almost trivial to implement, even if compared to standard GUI toolkits(but not browsers). Things that you have taken for granted (textboxes) require months to implement correctly. And you will still find corner cases, be it keystrokes(even without taking different operating systems into account).
It is easy to make it look 'flashy'. It is surprisingly difficult to make it look 'good'. We see some beautiful UIs in games and we think our applications can look the same. They really can't, not without great graphic designers. And no matter how great it looks, it will look 'alien' compared to other applications in the system.
If you need any kind of dynamic layout (which you normally do to handle resizing), you are now looking into an entire system and months of development.
Native OS controls usually have ways to hook up to them from external apps. Not so when you are drawing yourself. So that breaks accessibility but also any chance of automation.
All those drawbacks (and more I haven't touched) aside, there are benefits.
They can be incredibly fast, faster than native OS controls. Rendering a few quads is no problem for any GPU. You can have incredible interactivity(this needs to be explored more by non-Games), since rendering is so fast, faster than the older days of blitting stuff in framebuffers(and a lot of it can be handled by the GPU).
You can also implement transitions that are impossible to do in anything else (maybe except for web browsers). You can transition between 2D and 3D seamlessly.
For color blindness you can just apply a very simple shader over the whole thing.
I think there is a lot of potential hidden by this approach. Think of some the of GUIs imagined by works of science fiction. The main issue is that there's a lot of work to be done before you have anything functional and if you use something out of the box, it will be probably difficult to extend.
That said, I've considered taking this idea to an absurd level and write, say, a text editor on something like Unreal Engine and see how far into absurdity we can extend this idea.
What I'd think was the most demanding GUI application on my laptop also seemed the most responsive.
Something about using a level of abstraction at the wrong layer of the stack. I would expect GPU rendering to be used way deeper in the stack (OS level).