Leveraging Rust and the GPU to render user interfaces at 120 FPS
zed.dev
zed.dev
I'm surprised you did not look at "Dear ImGui", "Noesis", and "JUCE". All three of them are heavily used in gaming, are rather clean C++, use full GPU acceleration, and have visual editors available. Especially JUCE is used for A LOT of hard-realtime professional audio applications.
"When we started building Zed, arbitrary 2D graphics rendering on the GPU was still very much a research project."
What are you talking about? JUCE has had GPU-accelerated spline shapes and SVG animations since 2012?
BTW, I like the explanations for how they use SDFs for rendering basic primitives. But that technique looks an awful lot like the 2018 GPU renderer from KiCad ;) And lastly, that glyph atlas for font rendering is only 1 channel? KiCad uses a technique using RGB for gradients so that the rendered glyphs can be anti-aliased without accidentally rounding sharp corners. Overall, this reads to me like they did not do much research before starting, which is totally OK, but then they shouldn't say stuff like "did not exist" or "was still a research project".
You’d surprise a lot of people if you were right about it using GPU acceleration for its UI framework. It does have an OpenGL container that fits into its view object hierarchy if you want to write your own accelerated component, but the rest of the UI is pretty much standard event-loop -> platform-API dispatch. It’s accelerated where the underlying platform-API is accelerated but not in any kind of explicit or fine-tuned way. The focus has always been more on portability than performance, even on the audio side.
(And of course the high-priority audio processing runs in an entirely segregated context from the UI, so the performance characteristics of the two pieces are decoupled anyway.)
That JUCE is "heavily used in gaming" for starters is widly innacurate
I know it's tangential to the point that you're making, but JUCE was acquired by PACE in 2020.
There may be some people you know using JUCE for games, but it’s an odd choice.
Trying to figure out what you might mean, JUCE was involved with the BLOCKS SDK for Roli’s hardware peripherals, but again, that’s not really about gaming and definitely not about any familiar kind of gaming.
That said, I’m sure you’re relating the best information you have and very likely just transposed some detail by accident and are now caught in a relentless, exhausting online nitpick. We’ve all been there. If you figure out what it was, I’d love to hear it!
The kind of games some tint amount of people make with JUCE is like this: https://ankuznetsov.wixsite.com/jucegames and even those are few and far between.
In any case, nowhere even close to "widly used" in gaming. Nor moderately used, or even low use.
"Few and far between users of JUCE for games, and even at that almost all of them using it for hobby casual game projects" is an accurate description.
JUCE does dominate in audio VST/AU fx/instruments GUIs though...
Also, you can use OpenGL for GUI compositing, too, not only as a 3D context.
https://github.com/juce-framework/JUCE/blob/4e68af7fde8a0a64...
It uses CoreGraphics to render into a bitmap on the CPU, which is then uploaded into a Metal texture.
My 2 cents: it's nice to have smooth rendering in your editor, but I'm currently mostly using a Java-based IDE (IDEA family), and it's responsive enough for my taste. If I were to use the current prototype of their editor, I'm afraid the usage of fixed-width fonts all over the place (which I assume can be fixed, but may also be due to constraints of the UI framework?) would probably bother me more than the 120 FPS UI would impress me (assuming that I had a monitor that was fast enough, which I don't). On the plus side, it sounds like they support sub-pixel rendering - if that's the case, kudos to them!
And egui isn't tied to Apple proprietary frameworks.
Given that they don't talk about it in the README, I'm pretty sure they don't.
All the visualizations like the 3D scenes with point clouds etc. are rendered with our open-source in-house renderer ("re_renderer") which itself is then passing wgpu surfaces/commands to the egui wgpu backend that composites everything together then.
Zed is not constrained to use fixed-width fonts and supports all kinds of fonts, monospaced and variable-spaced alike (ligatures and contextual alternates included). Even though we use a glyph atlas, we rely on CoreText for shaping and rasterization (rendering sub pixel variants as well). In practice, this means that text is indistinguishable from a piece of text rendered by the operating system.
I'm not familiar at all with JUCE, but the state of the art in GPU-accelerated 2D graphics has dramatically improved over the past 10 years so I doubt what JUCE did in 2012 is really comparable.
This is known as a “multi-channel signed distance field”, or “msdf”.
BTW, the KiCad schematics are pretty text-heavy. You typically write down all the parameters and IDs of all the electrical components.
They are not using SDFs for the text, they render the glyphs at the specified font size directly.
It feels intuitive and that's why it's the de facto standard for building debug tooling. But this intuitiveness comes at a price! No matter how hard you try, this will always be slower and computationally heavy compared to retained mode.
Great for some stuff, terrible choice for some other.
immediate mode rendering and immediate mode gui are unrelated. immediate mode gui libraries generally dont use immediate mode rendering because its realy slow.
immediate mode gui tends to be faster in practice because you just naturally end up with less code. also its very easy to write a retained mode ui ontop of an immediate mode gui library.
It is like static linking, it was also there decades ago.
There are reasons why the world has moved on from those approaches.
And OTH, event-driven UIs have also been around as long as UIs exist, that makes them at least as 'outdated' as immediate mode UIs.
Some of them keep being rediscovered in endless loops of fashion.
The reason for this is often that the environment (mainly the hardware) has changed so much that it may make sense to look into discarded old ideas again. For instance, dynamic linking was extremely important in the age of slow floppy discs and when RAM was counted in kilobytes, but those environmental factors are no longer an issue, and the advantages of static linking outweigh the disadvantages of dynamic linking again.
And those “reasons” often SUCK; that is the point.
> Dear ImGui is designed to enable fast iterations and to empower programmers to create content creation tools and visualization / debug tools (as opposed to UI for the average end-user). It favors simplicity and productivity toward this goal and lacks certain features commonly found in more high-level libraries.
It's literally not designed for end user consumption.
I mostly stick with them because the flexibility and image quality feel worth it.
https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
It also does stroke handling on the CPU:
https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
Basically, this isn't really "GPU accelerated splines". It's a CPU coverage rasterizer with composting handled by the GPU.
CoreGraphicsContext::createPath will convert the CPU spline segments to CG spline segments which are then rasterized by CoreGraphics using Metal on the GPU.
https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
And on Windows 7 and up it'll use the hardware-accelerated Direct2D APIs:
https://github.com/juce-framework/JUCE/blob/2b16c1b94c90d0db...
CoreGraphics does not use the GPU.
Direct2D uses a approach which tesselates paths into triangles on the CPU. It is similar to the JUCE code in that the GPU is used for coverage, but it still does not natively render splines on the GPU.
And yes, I agree that JUCE will tesselate the splines into straight lines on the CPU. So technically, they are not fully rendered on GPU. But in practice, the slow rasterization is done on GPU while tesselation only has a negligible performance overhead. For example, I heard that UE4->UE5 removed the GPU tesselation support because it's usually easier to just DMA a new mesh from the CPU on-demand. I would treat that as a strong argument that CPU tesselation is practically never the bottleneck.
This class is part of a JUCE demo app, and you can read the source code to it if you want. [0] It uses CoreGraphics to render the graphics on the CPU, and then uploads it as a texture to the GPU so it can be used as a CAMetal layer. So, no, the graphics are still rendered on the CPU, with compositing being handled on the GPU.
I don't know why you're confidently saying something incorrect, it took me all of 2 minutes to find the source code and read it.
> For example, I heard that UE4->UE5 removed the GPU tesselation support
I know it's confusing, but GPU tessellation is a completely different thing. The word "tessellation" in graphics means "turn into triangles". In a 2D graphics context, we're turning splines and curves and 2D shapes into triangles. In a 3D graphics context, GPU tessellation refers to a control cage mesh which is adaptively subdivided. These two have nothing in common except that triangles come out the other side. I am not aware of anyone who has tried to use GPU tessellation to render 2D graphics.
GPU tessellation failed for a large number of reasons, but poor performance was one of them. So, you know, doing this sort of work efficiently on the GPU is still an open research problem. Just because it's not efficient to do it on the GPU does not mean the performance overhead is negligible. For rendering big complex vector graphics, tess overhead can easily outweigh rasterization overhead.
[0] https://github.com/juce-framework/JUCE/blob/4e68af7fde8a0a64...
I'd say we just have different definitions of GPU-accelerated. Your position appears to be "purist" in the sense that almost all processing needs to happen on the GPU. But for me, if mostly static data is generated on the CPU once and then cached on the GPU and then processed on the GPU in all performance-critical contexts, then I would call that "GPU-accelerated" even if parts of the data preparation happened on CPU. That's also why I brought up the UE. They prepare the data on the CPU, then send it over to the GPU so that rendering in the real-time loop can happen only on the GPU. But the fact that the CPU prepared the data does not make it not-"GPU-accelerated" for me.
That said, can we agree that the ID2D1GeometrySink that JUCE uses on Windows is GPU-accelerated? Surely the Direct2D/DirectX drivers will do stuff on the CPU, but Microsoft says "Direct2D is a hardware-accelerated, immediate-mode 2-D graphics API" and they show very smoothly antialiased splines here while mentioning "Multisample Antialiasing", a GPU feature:
https://learn.microsoft.com/en-us/windows/win32/direct2d/dir...
This is not to diminish the quality work that went into the library, but I passed over it for those and other reasons. I used nuklear for a bit since its typography was extensible but in the end it was too hard to overcome its other issues, so I use a custom mixed immediate/retained library now with rich text and accessibility support.
I'm always interested in learning about GUI toolkits that have accessibility support. What level of accessibility support does yours have? e.g. does it implement platform accessibility APIs, and if so, on which platforms? Is your library open source? If not, can I see it in action in any commercially available product? Thanks.
The way it works is that it has an immediate mode API (see https://github.com/sq/Libraries/blob/0ca01d949e3df5fabb1440d... for a simple example of the API) and then under the hood it uses a lightweight retained-mode graph, which means that when it's more convenient you can just write more traditional classes and components. In practice most of the UI I've written for my main project is a mix of both models, like for example an editor popup window uses IMGUI-style API while the items in a virtual listbox are represented by a small custom component.
It does have a full imgui-style layout engine under the hood instead of doing retained-mode layout, so it's able to fully re-generate layout from scratch every frame, and to minimize page tearing there is a system where components can request a second relayout pass (typically used for things like text autosize).
Here's a more complex mixed-mode example from one of my development tools: https://gist.github.com/kg/6a6ba42d5019b546858a2b18751de019
Almost all of the on-screen elements in this footage are either immediate-mode or retained-mode UI using the library: https://www.youtube.com/watch?v=ey3FtFWxbhA
Is this stuff written down somewhere other than the code?
I use KiCad all the time and certainly never noticed this.
What causes user interfaces to hick up is that it's too easy to do stuff in the main UI thread. First it doesn't matter but stuff does accumulate, and eventually the UI begins to freeze briefly for example, after you press a button. The user interface gets intermingled with the program logic, and the execution of the program will visibly relay its operations to the user.
It would be very much possible to keep the user interface running in a thread, dedicated to a single CPU on a priority task, updating at vsync rate as soon as there are dirty areas in the window, merely sending UI events to the processing thread, and doing absolutely nothing more. This is closer to how games work: the rendering thread does rendering and there are other, more slow-paced threads running physics, simulation, and game logic at a suitable pace. With games it's obvious because rendering is hard and it needs to be fast so anything else that might slow down rendering must be moved away but UIs shouldn't be any different. An instantly reacting UI feels natural to a human, one that takes its time to act will slow down the brain.
But you don't need a GPU for that.
You could have some wins around scrolling and any animations that you can commit ahead of time
I do think that they will need to do something similar with a higher level language support for extensions. Could probably piggy-back the Deno work for using V8/JS/TS for extensions similar, maybe even compatible with VS Code extensions. It's a massive user space, but seeing the core feature set in another editor could be really nice.
For me the killer feature of VS Code is really the integrated terminal. Not having to switch windows, or leave the app to run a quick command is really big imo.
The latency that matters for a text editor is how long it takes for a new keypress to show up. And that's usually a 32x32 pixel area or less.
It's just like a video game. A consistent 60fps is much better than an average frame rate of 120fps that drops to 15fps when the shooting starts and things get blown up. You spend all the time optimizing for the worst case where all your caches get invalidated at the same time.
[1] https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-fove...
See for example https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_buff...
And https://registry.khronos.org/EGL/extensions/KHR/EGL_KHR_part...
The reality is that moving things off the UI thread is always a calculated engineering decision. It can be worth it, but it complicates the code and should only be done if there's a realistic chance of the user noticing.
Nowadays I still regularly see applications with (temporarily) frozen windows and I just don't understand how that's possible. When I was developing my winforms apps, anything that would do more than perfectly predictable UI manipulations was run on a background thread (in a task), would be forbidden from starting twice at the same time and only updated the UI when done.
Probably kind of like this: https://stackoverflow.com/a/36107907/2306536
I haven't developed C# for almost a decade, so I probably didn't even have access to async/await back then.
1) Make anything that accesses a resource over the network, in a database, or in the filesystem async. So basically any time you make an HTTP request, execute a SQL query, or open a file etc. Otherwise you should generally stick to synchronous since async code is actually slightly slower than synchronous code. An exception to this might be if you're doing some heavy computation that's going to take a while and don't want it blocking the UI thread
2) Async is viral. If you have a method that calls an async method, you need to make the calling method async too. You need to then make its own calling method async as well all the way back to Main()
3) Never use async void as a return type, with the sole notable exception to this rule being event handlers. Use async Task instead as the return type (or async Task<T> if it returns a value)
4) Always call async code with the "await SomeAsyncFunction()" syntax. Calling it other ways like "SomeAsyncFunction().Result" will eliminate most of the benefits of using async in the first place since it is blocking, can cause deadlocks in some cases, etc.
Follow those 4 rules and you don't really even need to understand how async works under the hood, though personally I'd recommend learning it as it will help you a lot when you run into issues with edge cases.
No we didn't. We had nothing close to that in the 90s.
Typing was responsive, but that's not anywhere close to 60fps and a very small region to update to boot (120wpm = 600 keys/minute = 10 keys/s, or 10fps). Scrolling benefits from 60fps+, but in the 90s scrolling jumped by lines at a time because it couldn't do anything better.
You need a GPU to keep up with smooth scrolling at modern resolutions. But this also shouldn't be a surprise. What the article talks about is bog standard stuff for all current UI toolkits. It's what mobile platforms like Android have been doing for a decade now.
Was amazed at how good my latest iPhone feels with 120hz scrolling. It's like magic.
That would be my advice to anyone making a gpu-accelerated ui library in 2023: Try to support accessibility, and even better: make it a first class citizen.
ARIA is a good start, and screen readers built into the OS are a good start.
Moreover, major OSes have accessibility APIs that screen readers will use:
- MacOS https://developer.apple.com/library/archive/documentation/Ac...
- Windows: https://learn.microsoft.com/en-us/windows/apps/develop/acces...
Edit: BTW, I've been in the assistive technology industry a while, particularly on Windows. Feel free to ask me anything.
Our team is developing voice control for a website. This is mainly requested by sighted users who want to use the site hands-free. But we also have an accessibility mode for better screen reader support.
We think voice control might be appreciated by screen reader users too, but we aren't sure how well it would work with a screen reader.
Are there common pitfalls we should be wary of?
One thing we're worried about is that the voice from the screen reader might interfere with the detection of the user's voice.
I know general dictation and voice control software already exists, so my initial assumption that screen reader users would benefit might be wrong. If the existing tools are good enough, perhaps this whole question is moot.
In general, you should assume that the user already has whatever assistive technologies they need, and you don't need to provide your own, just make the content and UI accessible using semantic HTML and (if needed) ARIA. Providing your own screen reader, for example, would definitely be a mistake. The same should ideally also hold for voice control, but apparently you're actually getting some demand for that.
I love immediate feedback but getting it ranges from hard to neigh impossible. E.g. I have a complex Emacs setup for rendering Pikchr diagrams, but there are a lot of problems to solve from diagram conception to the end result, so I thought, hey, why not make my own cool RT editor - in Rust obviously.
Unfortunately I learned that GUIs are though problem especially if idea is hobby-based so there's only one developer inside. Ultra responsive GUIs cool, I have a prototype in egui (not sure if that's as fast as Zed's premise but feels fast nonetheless) and yet it doesn't support multiple windows, which I wanted to have.
120 FPS with direct rendering sounds AWESOME just for sake of it, but I believe that for the end-user layout will be more important than refresh rate, and that's different beast to tame.
Personally I "almost" settled for Dioxus (shameless plug: [1], there's link to YT video) and I'm quite happy with it. Having editor in WebView feels really quirky though (e.g. no textareas, I'm intercepting key events and rendering in div glyph-by-glyph directly).
You're right that rendering is only part of the story. To stay within the ~8ms frame budget, however, every little bit counts. Maintaining application state, layout, painting, and finally pushing pixels to screen, all need to be as performant as they can be.
For layout specifically we're using an approach inspired by Flutter, which lets us avoid complex algorithms but still have a lot of flexibility in the way elements can be positioned and produce a rich graphical experience.
Thanks for reading and commenting!
Rebuilding whole app to ensure that some box doesn't get cut off ruins development experience, especially for big apps.
Kudos to you guys, I hope you'll make Zed extensible, so that instead of writing my own editor I can use yours ;-)
I'd like to see "gedit", for Linux, fixed. It can stall on large files, and, in long edit sessions, will sometimes mess up the file name in the tab. Or "notepad++" for Linux.
Edit: Yay, it's a text editor. What happened to only redrawing the line that's being edited and the status indicators?
Zed and GPUI use energy very judiciously and only perform updates when needed. The idea is that we can render the whole application within ~8ms, and that shows everywhere: from typing a letter to scrolling up and down in a buffer. However, if the editor is sitting there idle, we won't waste precious CPU cycles.
Thanks for the feedback!
I mean, the win16 api from ages ago had support for invalidating specific screen regions etc. It probably got lost somewhere in the transition to javascript...
For example, their editor has tabs for editor buffers. Cocoa has a static tabbed widget, which has wrong look and odd UX for this. Cocoa also has a tabbed window type, which isn't a widget you can control. I imagine it'd be hard to abstract that away to work consistently with how Windows does tabbed views. I also haven't seen Windows' tabs being draggable, so that would probably need special DIY solution for Windows which Cocoa tabbed windows don't need.
Anyway, I think native UI toolkits are dying. For most people the Web is their most familiar "toolkit" now, and native platforms instead of fighting that back with clear consistent design, went for flat design and multiple half-assed redesigns that messed up all remaining expectations of how "native" looks and feels.
For very contrived example, the cross platform settings panel widget exposes the business API required for settings in general, while the platform specific code takes care of using the host platform concepts to display and manage application specific settings.
While it is more work than lowest common denominator approach, it is still much less than re-inventing the wheel.
As proven by mobile OS platforms, there are still hope for native toolkits.
Lets see how much long the Web will hold, now that the revenge of plugins is here thanks WASM.
But the HW accelerated brag is pointeless. Even if they manage to squeeze some extra performance over native toolkits, that is not necessarely going to matter in the grand scheme of things. Drawing the UI is never the bottlneck in a text editor...
> Drawing the UI is never the bottlneck in a text editor...
WinRT has gone through multiple reboots, who knows what will happen still.
Better use the existing Win32/COM stuff.
After our past experience with Atom, getting the plugin system right is a top priority for the editor.
The thought of cross compatibility with VSCode plugins definitely crossed our mind and it's not out of the question, although our current plan is to initially support plugins using WASM.
Another option in this front could be Flutter if write-once run-everywhere is a need for the project. Another advantage is that it's not only GPU accelerated but it's also retained mode.
The article is talking about the same generic hybrid GPU-accelerated rendering architecture that everything uses. Seemingly the only "new" part is "in Rust!"
Which is kind of a shame. But it's the result of years of product management neglect as well as the pull away from desktop UIs to web UIs.
Native AOT still can't compile WinUI, while C++/WinRT is like doing ATL in 2000, while bug issues grow exponentially.
Only WinDev themselves, and WinUI MVPs, can still believe this is going somewhere.
The rest of us have moved on.
Is their GPUI library open source?
I'm surprised nobody pointed out lite/litexl here either it's rendering of ui is very similar (although fonts are via a texture; like a game would) and doesn't focus overly on the GPU but optimises those paths like games circa directx9/opengl 1.3
There are great details of the approach taken with lite at https://rxi.github.io
Lite-xl might have evolved the renderer but the code here is very consumable for me.
It should be noted that the main person behind Zed is Nathan Sobo, who created Atom while he was at Github, which is the basis of Visual Studio Code today.
As such, I have high hopes Zed will be a much faster version of Visual Studio Code and am excited to see what him & his team make.
IME the main theme with achieving high performance not just in games, and not just in rendering, is to avoid frequent 'context switches' and instead queue/batch a lot of work (e.g. all rendering commands for one frame) and then submit all this work at once "to the other side" in a single call. This is not just how modern 3D APIs work, but is the same basic idea in 'miracle APIs' like io_uring
This takes care of the 'throughput problem', but it can easily lead to a 'latency problem' (if the work to be done needs to travel through several such queues which are 'pipelined' together).
Curious if you guys have thought about VR / AR possibilities with GPUI?
Would probably just be stupid gimmick, but I want to see somebody do it :-D
Windows uses GPU-based rendering in most/all of their GUI frameworks. Chrome also makes heavy use of hardware acceleration. Rendering performance isn't left to the window manager. If you're making your own GUI this is exactly the kind of stuff you need to do to make it fast.
Second, forgive a naive question since I know nothing about graphics, but would the method described in the article perform better than Alacritty + Neovim?
I was really excited when I saw that demo. Why didn't this turn into a final product that people could use?
I suspect a lot of time is likely to be spent on the CPU side updating vertex and other data and pushing it to the GPU so it would be useful to have some more detail on how they are handling that.
(Also GPUI seems to be retained, so completely unrelated.)
https://nick-lab.gs.washington.edu/java/jdk1.5b/guide/2d/spe...
Similarly, if you're dragging something around with your finger, 120fps keeps that thing closer to the finger. Just improves the UX.
Could you please point me here as to what I am missing ?
And it's not just about electricity cost and heat stress, it will conflict with everything else that requires the gpu to do stuff, including watching 4k videos on the second monitor, which does have a legitimate case for requiring hardware acceleration since they move a lot of data 60 times per second, and your editor doesn't.
And the limited resource is not the gpu itself but the nearby onboard memory is a scarce resource on its own. I'd be real mad at a software that prevents me to multitask.
Additionally, using your GPU doesn't mean locking your GPU on that task. Your 4K video most likely isn't taking up 100% of GPU time, nor 100% of the nearby onboard memory. Everyone already gets a share of that time.
Chasing 120fps rendering here is the problem.
I’m honestly thankful for this bit of lying. Windows 8 and below had horrible, horrible issues with fullscreen programs, especially games - they’d change your screen resolution and thus move and resize all your other windows, they’d freeze for ten seconds when you tried to alt-tab out, and they’d frequently just crash when you switched back to them. Those problems are essentially nonexistent nowadays, and it’s worth the small performance cost that comes from going through the compositor.
But yes, alt-tabbing Source games was an exercise in patience.
Then again the issue is the focus of rendering at 120fps.the rendering context should be paused between the vast nothingness that happens in between user keystrokes, that happen at something like 4kp, and in burst.
The insistence on 120fps is the issue here, not the rendering pipeline.
All I see is them saying they can, not that they are.