Immediate Mode GUI
behindthepixels.io
behindthepixels.io
Immediate mode UIs come up every now and then, especially in games, and for relatively simple things they're a neat idea (imagine not retaining any extra view state!) But - and there's a big "but" coming up - it does not scale to complex and stateful UIs.
The problem is that, for games that have complex needs - think simulation and strategy games, not racing or arcade - complex UIs are needed to express complex relationships between underlying data and the relevant UX flows. But how do you do that in IMGUI? This information about UI state, layout, etc. is not in the UI anymore, but it has to live somewhere - so you end up having to build your own layout and UI management stuff, and basically duplicate what a RMGUI library would offer, but now with less generality and potentially more bugs.
For the probably best known example, Unity used to have an IMGUI [1] but they've actively moved away from it. It's still there in vestigial form, the API is not removed, but game developers don't really use it.
[1] https://docs.unity3d.com/Manual/GUIScriptingGuide.html
PS. And there's also the lack of visual layout tools, which is directly related to IMGUI's reliance on code to produce layout.
It's true that the newer retained mode GUI has supplanted the older IMGUI for "game UI" (e.g. what users interact with when playing an actual game built with Unity). On the other hand, the Unity editor itself is still built entirely with their immediate-mode GUI library, and I haven't heard of any plan to move away from that. Building "editor-like" tools (including custom user tools) is still significantly easier to accomplish using immediate-mode libraries.
https://blogs.unity3d.com/2019/04/23/whats-new-with-uielemen...
Properly object oriented retained mode GUIs let you subclass components to implement custom views and event handling, and develop and publish reusable prefabs and libraries of pre-configured, stylized, specialized component.
But immediate mode GUI lets you compose very well. It's a bottoms up technique of programming, where instead of following a framework established by the library author, you are given low level building blocks. You will find it hard to mix and match two retained mode GUI libraries unless they all follow a common set of standard apis (e.g., java swing). But with immediate mode GUI, it's actually easy to mix and match (as the immedate mode gui has no need to know what framework you're working under nor need to make any assumptions about threading or api).
I've implemented various adaptor wrappers for composing different retained mode API's myself, including wrapping The NeWS Toolkit OPEN LOOK widgets in HyperLook widgets, which I used for the first Unix port of SimCity to HyperLook/NeWS. (The OPEN LOOK pin-up menus and pie menus and buttons and sliders are implemented in TNT, but I wrapped them for HyperLook (which has its own persistence system and class hierarchy), so you can dynamically create and edit them at runtime with property sheets and script editors, ala HyperCard.)
https://medium.com/@donhopkins/hyperlook-nee-hypernews-nee-g...
For X11/NeWS at Sun, we even made an ICCCM X11 window manager that wrapped X11 windows in NeWS tabbed window frames with pie menus! (The X11 window manager was written in object oriented "retained mode" PostScript code, subclassing standard and custom NeWS window and menu widgets, like tabbed window frames and pie menus. While PostScript has an immediate mode drawing API, it was driven by extensible "retained mode" widget instances and classes represented by PostScript dictionaries, pushed on the dict stack.)
Open Window Manager PostScript source: https://www.donhopkins.com/home/archive/NeWS/owm.ps.txt
Tabbed window extension: https://donhopkins.com/home/archive/NeWS/win/tab.ps.txt
Discussion of problems with NeWS: https://news.ycombinator.com/item?id=15327339
Political discussion from the Window System Wars, about Sun's problems merging X11 and NeWS: https://donhopkins.com/home/archive/NeWS/sevans.txt
More recently (and more successfully), I've wrapped HTML components and floated them over OpenLaszlo/Flash (not so recently) and Unity3D (quite recently) retained mode user interface layouts in the web browser.
For example, I'm embedding an ACE code editor in a Unity3D retained mode user interface running in WebGL, that falls back to a regular TextMeshPro input field on other builds. There's no way you could do that practically with an immediate mode API.
WebGLAce_TMP_InputField.cs (C# retained mode GUI TMP_InputField subclass): https://github.com/SimHacker/UnityJS/blob/master/Libraries/W...
WebGLAce.cs (C# P/Invoke Unity extension wrapper): https://github.com/SimHacker/UnityJS/blob/master/Libraries/W...
WebGLAce.jslib (JavaScript library Unity extension, compiled by Unity to WebAssembly, converts between C#/WebAssembly and JavaScript data types and thunks messages back and forth between the browser and the Unity3D app): https://github.com/SimHacker/UnityJS/blob/master/Libraries/W...
WebGLAce.jss (ACE code editor adaptor, pure JS source loaded by the browser, as well as ACE JS libraries themselves): https://github.com/SimHacker/UnityJS/blob/master/Libraries/W...
Notice how WebGLAce_TMP_InputField simply subclasses the standard TextMeshPro TMP_InputField, and adds some more hidden state like the editorID and editorConfigScript of the corresponding ACE code editor running in the web browser.
All that is hidden from the code using it, so it doesn't need to come up with another place to store that editorID and editorConfigScript, or even know if it's getting an Ace editor under WebGL or falling back to TextMeshPro on other builds. That's what I mean by leveraging OOP to subclass existing components and hide internal state, which you can't do in immediate mode.
(My plan for non-WebGL builds that aren't already running inside a web browser, is to create an embedded web browser just to run the Ace code editor, and the C# WebGLAce_TMP_InputField API will remain the same, and it will just work transparently on different platforms. I want to support live coding Unity apps in JavaScript, but there just isn't anything for Unity that approaches ACE for editing code, so it's worth going through all the trouble to make an adaptor.)
How would you write a functional extension for Unity3D's old immediate mode GUI that let you embed an ACE code editor in a Unity WebGL app? What would the API and state management even look like? How could you make it platform independent?
And even if you solved all of those problems with an immediate mode API, by its very nature it still wouldn't enable you to build GUIs in the interface editor or store them in prefabs, and you'd still have to recompile you app (which can take half an hour with Unity) every time you wanted to tweak the user interface (which is why I like programming Unity apps in JavaScript as much as possible).
UnityJS Unity3D / JavaScript bridge architecture: https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy...
UnityJS pie menu component for Unity3D, written in JavaScript, drawn with canvas, using TextMesh Pro labels: https://github.com/SimHacker/UnityJS/blob/master/Libraries/U...
Uses a Unity3D pie menu tracker to translate low level input to high level events, written in C#: https://github.com/SimHacker/UnityJS/blob/master/Libraries/U...
Of course you can take it too far, trying to please everyone by combining the worst of all possible systems and then giving it a ridiculous sounding name (see "MoOLIT"), but we're discussing whether it's possible and practical, not whether it's desirable. And even if it's not desirable, it ends up being necessary.
https://en.wikipedia.org/wiki/MoOLIT
http://nova.polymtl.ca/~coyote/open-look/01-general/faq-doc-...
Embedding object oriented "retained mode" widgets with different APIs inside of each other is old hat and common for backwards compatibility. Whenever you write a new GUI toolkit, embedding widgets from the last toolkit is one of the first things you do (including recursively embedding widgets from the toolkit-before-last).
Concrete example: Microsoft lets you embed old fashioned OLE controls in Windows Forms / Presentation Foundation applications, which might be implemented in MFC themselves. And MFC is all about embedding old Win32 widgets implemented in C, and newer OLE components implemented in Visual Basic or whatever, in venerable C++ MFC user interfaces. Say what you want about how much Win32/MFC/OLE/WF/WPF sucks, and I'll wholeheartedly agree, but if you're running Windows, you probably have widgets on your screen using several different retained mode APIs embedded more than two levels deep right now.
The problem with immediate mode that is you have to come up with somewhere to store and retrieve any values or state required on a case-by-case basis (including the edited value itself, and other view state like scrollbar state for text editors, etc), and that tightly couples your component function with whatever's using it, so they're not very reusable or simple. With OOP, the object has its own place to store that stuff, which is cleanly decoupled from whatever's using the component.
Then there's the issue of event handlers. Some user interface components just aren't so simple that they only have one true/false return value like buttons. Text editors can notify on value change, end edit, select, deselect, etc. And Unity's retained mode GUI supports persistent event handlers that let designers hook up events with methods of objects with parameters, without writing or recompiling any code.
And there's also a lot more to layout that you can easily express with an immediate mode GUI. Sometimes you have to measure a bunch of things and align them in various ways, like grids or flex layouts, and adapt to the screen size and other constraints (i.e. responsive design), and that's really hard to do with immediate mode, while retain mode has a rich set of layout components you can use and extend. And there's nothing like XCode's layout constraints for immediate mode.
A perfect example is TextMesh Pro's text editor. It has so many different (and useful) features and parameters and callbacks and ways to configure it, and it keeps so much state in order to redisplay efficiently with the fewest number of draw calls and reformatting, that it would be extremely impractical and inefficient and unwieldy for it to use an immediate mode API. Especially with C# positional arguments instead of Pythonic or Lisp-like optional unordered keyword arguments.
When you encapsulate your immediate mode layout in a class, or write an engine that interprets (i.e. JSON) data to drive the immediate mode API, or use reflection in your wrapper functions to dispatch event callbacks and bind your widget data to your model data, you're just rolling your own informally-specified, bug-ridden, slow implementation of half of retained mode. (See Greenspun's tenth rule.)
I'd push that yes, you do have to maintain your state externally in an immediate mode UI, but when you have a lot of "state" that's really just a function of other state, that's exactly what you want.
It’s a fascinating hybrid - rather than “BYO State” where the library does nothing but layout and draw, or “BYO hierarchy” where you need to retain instantiated components, React is “hierarchy tracking included” - all the benefits of persistent components, but tracked by the renderer rather than the caller. It’s an evolutionary step that I think desktop programming would do well to adopt and continue to iterate on.
I think the following is an interesting example here: your application state is some type, Config which would correspond to some boring components, e.g. text boxes and radio buttons, but the interface you want to present looks like:
[X] default settings
[ ] default settings
...
Maybe Config
Bool * Config
(mutable) Bool * Config * (mutable) Config
I suppose this is like the very early web dev style of having no direct state and directly reading values from controls, but at least in this world one has control over how the view state is managed and represented so it can be easier to avoid getting into invalid states.
Immediate mode is about the interface to rendering the UI, not necessarily about the implementation (although some people conflate these). So your UI elements can be stateful and retain their own state, but they can be released once they are no longer rendered in the current frame.
I'd rather manage the state and relationships myself.
This is why every tells you to use MVC or some variant that separates out the UI from the data. IMGUI lets you bend the UI to the data instead but the real lesson is to decouple the view and the model.
Which is why the original Smalltalk people -- who invented MVC -- mostly switched to Self's Morphic framework in their later projects (Squeak, etc.)
Typically we build GUIs using event driven callbacks. This causes problems in managing life times, dynamic layouts, threading issues etc. The idea behind immediate mode GUI is that you get a frame and in each frame you create/update layout, check for user actions - just like you do in render frame in game loop. So there are no call backs and you don't have to worry about issues arising from event-driven code.
If anything, this is actually a lot like Web 1.0 programming: server side rendering, often using templates that end up looking a lot like a DSL. Each page view is rendered based on those templates against data fetched in advance of the output.
You can get pretty deep with that kind of thing, though of course the page refresh time was what killed this otherwise fairly nice style of writing an application. Users expected actions like sorting a column to be as fast in the browser as it was in a Windows 95-era file explorer or Excel, and it simply wasn't happening.
(Imagine if first-class client-side-interactive UI components like a sortable table, tabbed panel, etc. like what XUL provides had been in HTML in the 90s. We wouldn't need half the Javascript bullshit people have to use now to replicate that functionality half-assedly...)
https://pbrfrat.com/post/imgui_in_browser.html
For modal demo check "Popups & modal windows" -> "Modals" example.
This is how it looks in C++ code: https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#...
https://greggman.github.io/doodles/glfw-imgui/out/glfw-imgui...
In the "ImGUI Demo" window start expanding the collapsed items or from the menus pick "Examples"
Note that no effort has been spent to actually make the demo web browser friendly so there are rough edges.
Of course I know what a GUI is. I've been building them for decades, and building tools that other developers have used to build GUIs. (I was one of the authors of the "visual" part of Visual Basic.)
But I have never before seen the terms "immediate mode GUI" or "retained mode GUI". And I'm pretty sure there are many developers like me who will have no idea what you're talking about. It takes only a moment to add a quick definition or a link the first time you mention a specialized term, and your readers will definitely appreciate it. Thanks!
(Edited to remove fluff and make the essential point more clear.)
In 1995 Microsoft acquired a company called RenderMorphics and turned their Reality Lab [1] API into "Direct3D". The first two versions to ship from Microsoft had both retained and immediate mode APIs, but they dropped the RenderMorphics-based API due to lack of adoption.
glBegin(GL_TRIANGLES);
glVertex3f(...);
glVertex3f(...);
glVertex3f(...);
glEnd();In fact for 3D rendering, retained mode was touted as a the new way forward, there was even a standard predating OpenGL called PHIGS, a retained mode system.. OpenGL was the worse is better (also PHIGS had many limitations basically limiting its use to CAD) backlash promoting its immediate mode in the early 90s.
It's the same principle as writing readable, self explanatory code. You write your article once, but you hope that many thousands of of people, with all different levels of experience, will read it. If you can take a few minutes to save thousands of readers a minute or two each, that is a big win.
If you really want to do it right, you can link to those Wikipedia articles when you introduce the terms. After all, your reader may be on a mobile device on the bus, where it is easier to tap a link than to copy and paste or retype the term to find out what it means.
(Here's hoping that everyone actually edits their blog post before posting.)
I did a Google search but there weren't any clear definitions on the first page or so.
I did search for the Wikipedia article and here's what I got:
"Immediate Mode GUI is a GUI where the event processing is directly controlled by the user. The key is the aspect of retained (RMGUI) vs non-retained UI (IMGUI). In the latter, the user code is holding on its own natural data, in the former it is stored in the GUI system, often hidden from the user code. The immediate mode GUI widget toolkits are more direct and thus less complex, thus they are a good choice for those who want a simple but easily changeable and extendable GUI toolkit, they are usually generic, open source and cross-platform. The original Immediate Mode GUI toolkit was imgui[1] which is based on OpenGL. The idea was popularized by Casey Muratori. There are others such as nuklear[2] with backends for several graphics libraries, kiss_sdl[3] for SDL (Simple DirectMedia Layer) or dear imgui[4] written in C++ (here the jvm port[5])."
First, the quality of this writing isn't great, to the point where it makes it harder to read.
I think that the above definition makes sense if you already know what Immediate Mode is, but it was pretty opaque on my first read (partly because I misunderstood "user" in the first sentence to mean "the customer sitting in front of the computer and using your IMGUI program", instead of "other code that uses the IMGUI").
The definition now makes sense to me after reading other posts here explaining that this was a Direct3D thing.
So yeah - I can see the arguments that we're not the target audience (and I understand that the person who posted this to HN probably isn't the author), but I think that Stratoscope's point that articles posted to the general HN audience would benefit from short, clear definitions (or links to definitions) in the articles is spot-on.
Also, props to Stratoscope for their calm, constructive replies in the face of a lot of disagreement! If we all did this (and I include myself here for sure) the Internet would be a better place.
We may be a technical audience here on HN, but I wager that the vast majority of HN readers have never heard these terms before, just as I never had.
Based on what kyberias said in a sibling comment, it sounds like these terms will be familiar to those who have written DirectX and perhaps other GPU code, but for the rest of us they are esoteric terms.
It is so easy to add a link when you first mention a term like this. Why not do that?
I am offering writing advice that can benefit anyone writing on a technical topic.
You may have an idea of who your intended audience is, but when you put something out on the internet, you don't know where it will end up and who will read it.
It only takes a few minutes to link terms to their definitions or to write a very short definition.
Your expected audience may not need it, but your unexpected audience will definitely appreciate it.
I think it is perfectly OK to have an intended audience and ignore an unintended audience if catering for the unintended audience dumbs your writing down for your intended audience.
It is good writing advice to not clutter your writing with needless explanations.
So wouldn't linking to definitions or references make both of us happy? If you're in the target audience you can ignore them, and if you're someone more ignorant, like me, you can click the link to learn more. Seems like a good thing all around.
At the company I work for, so many things I need to know are internal and so impossible to Google, and even asking somebody usually yields a less than thorough answer. I have to build a picture of how the whole thing works in my head over time. Then I become the resource everybody who is in the position I was before hounds.
Having to Google a term because the author didn't think to include a link is roughly a 0.01 on the frustration meter compared to my current job.
I get the desire for easy and flat interlinking of information. But we need to look at the people who do the work to make that happen as bearers of gifts, not sources of misery. Be the change you want to see in the world, don't just demand others do it for you. Google is your friend.
Most likely he's writing it for an audience that knows very well the difference between immediate/retained modes. I mean, he's writing it on a blog that's all about graphics programming, game engines, etc. So it would be a bad move for him to define terms common to that field in every article.
Given that, this critique is misplaced.
Like, c'mon guys.
The terminology he uses is in common use.
The article starts with an example, and then goes on to explain the difference between immediate mode GUI and retained mode GUI is. It is well written and easy to understand article.
What you are asking for is a TLDR at the top of the article. Sure, that would be nice, but on the other hand I don't see anything wrong with writing a blog post that requires reading more than 50 words to fully understand the topic.
Ha, you got me there! (I know the HN guidelines recommend to not suggest that someone hasn't read the article, but in this case I'm fair game.)
When I first skimmed the article, I only had a few minutes between tasks at home, so I didn't catch all the details. But since my initial comment stirred up some controversy, I figured I'd better take a closer read. (Well, still kind of halfway skimming since it is getting rather late here.)
In the next to the last paragraph was the clue that made it connect for me: "The OnGUI of Unity Engine actually uses IMGUI mode."
Aha! So this "immediate mode GUI" business is like OnGUI? Where you draw everything from scratch on every frame, instead of the engine remembering stuff for you?
That's when it all made sense, since I had done a bunch of Unity work 4-5 years ago and some of it involved OnGUI().
60654's comment led me to the current OnGUI documentation [1], where they even call it "Immediate Mode GUI (IMGUI)".
I'm pretty sure they didn't use the IMGUI terminology when I was working in Unity, it was just OnGui() with no other special name. Otherwise I would have remembered "immediate mode".
But it's possible that my memory is not as good as I hope it is!
Please understand that I was not intending to be hard on the author or criticize them. I am offering some writing advice that I hope could be useful for anyone writing technical articles like this. Of course that may just be arrogance on my part. ;-)
When I don't know a term I google it. I don't assume the entire world revolves around me and that it's the author job to make their article more verbose because everyone reading it might not know every term.
Here's are galleries of some from one of the more popular libraries
https://github.com/ocornut/imgui/issues/2265 https://github.com/ocornut/imgui/issues/1902 https://github.com/ocornut/imgui/issues/1607 https://github.com/ocornut/imgui/issues/1269
https://user-images.githubusercontent.com/16171743/50758310-...
p.s. I did get a chuckle from your deleted comment that suggested I was living in a cave. After the heat wave we've had here the last couple of days, I wish I were living in a nice cool cave! ;-)
I would argue that "immediate mode GUI" is an extremely specialised term.
Of course you didn't, Qt is retained.
It's a bit like imperative programming vs functional programming. Before functional programming gained any significant traction, imperative programming was just "programming".
> what books could I have bought but didn't?
Possibly none at all. Immediate mode anything is mostly a game dev thing, and what happens in game dev tends to stay in game dev. They don't seem to document their practices as much as other fields do.
Hence my suspicion that the authors of your books didn't even know about IMGUI. They just knew about "the" way to do GUI, which happens to involved retained state.
BTW, the code in Qt that draws rotated/sheared text/pixmaps was influenced in part by a book or chapter by Michael Abrash, as well as by articles in either Graphics Gems or conference papers (memory's fallible I'm afraid).
I agree that the docs could have been improved by providing some sort of definition. But I think something as simple as a link to wikipedia could have been enough.
I also agree that the intended target audience will most likely already be aware of immediate vs retained. Adding the aforementioned link would not have hurt said audience anyway.
You must have understood the difference between the VB shape control and the VB drawing commands then, if you wrote these things?
The shape controls were retained and the drawing commands immediate. You could move the shape controls around, but not the drawn shapes after they've been drawn.
The double buffer option on forms (can’t remember the name) was a workaround for a limitation of the immediate drawing commands, where they disappeared when the window was covered and revealed.
Even the Win32 API documentation uses the terms https://docs.microsoft.com/en-us/windows/desktop/learnwin32/.... Presumably you used that documentation to build VB?
What term did you use back then if not immediate and retained?
But you ask an honest question, so I will give you an honest answer: I have no clue!
We didn't have any terminology like immediate or retained. What we did have was the early Windows API with update regions and WM_PAINT. There was no desktop compositing, so if a window was partially or fully covered and then uncovered, the newly-uncovered area would be invalidated and the window proc would receive a WM_PAINT message to notify it to repaint that area. Could that be related to what you're referring to?
I should explain that our team didn't write all of the "visual" part of VB. We built a product called Ruby (no relation to the programming language) that was originally going to be a user-customizable desktop for Windows 3.0, replacing ProgMan and FileMan. After we turned it over to Microsoft, they decided to combine Ruby with Basic to make a full programming environment, and of course they made many changes and additions along the way.
There are many fun stories I could tell from that era, like the true origin of the term "fire an event", but I'm afraid none of them will involve immediate or retained mode GUIs... :-)
Update after seeing your subsequent edit:
> Even the Win32 API documentation uses the terms https://docs.microsoft.com/en-us/windows/desktop/learnwin32/... Presumably you used that documentation to build VB?
(and from the link)
> Graphics APIs can be divided into retained-mode APIs and immediate-mode APIs. Direct2D is an immediate-mode API. Windows Presentation Foundation (WPF) is an example of a retained-mode API.
No, we did not use that documentation. There was no such thing as Win32 at the time, and certainly no Direct2D or WPF. How could we have used Win32 docs? We were just 16-bit Windows programmers, we weren't time travelers!
Ye that's it.
So if you drew a shape onto a normal window, it went into the screen buffer, and would be erased if another window covered that part of the screen. If you wanted the shape to be visible again, you had to draw it again when the WM_PAINT message came. VB didn't remember that you used a drawing command and didn't keep the original details of the shape stored anywhere. That's immediate.
If you used one of the VB drawing controls to draw the same shape, then when the window was revealed after being covered, the shape control, an object in memory, was still there in order for the shape to be automatically redrawn by the VB runtime on the WM_PAINT message itself.
A half-way was to make the shape drawing commands write to a separate GDI (the Win32 graphics API) surface, which was then in a sort of retained mode itself, redrawn to the screen whenever the window was revealed. That was the double-buffer option on forms.
> There was no such thing as Win32 at the time
Well you know what I mean. GDI was part of the 16 bit API as well, possibly called WinG back then I think? Obviously they've updated the documentation as they've gone. But Windows included both retained and immediate components back then.
We didn't use WinG [1] - it came along years later, and I think only on Win32, not Win16? We did use GDI, of course, it was there from the very beginning, but there wasn't anything resembling retained mode back then. Just WM_PAINT and update regions.
We actually didn't even worry much about having controls automatically redraw themselves. For the most part, we were just wrapping standard Windows controls or combinations of them, and we let those controls respond to their own WM_PAINT messages as usual.
Of course MS undoubtedly made many improvements after we finished our part of the project. Fun times.
On second thought, that's not right either. Immediate mode is like Unity's OnGUI where you draw everything on every frame.
On Win16, you only got a WM_PAINT when needed, and it came with an update region of whatever had been invalidated since your last WM_PAINT. Many apps would optimize by just repainting the update region.
The Mac worked the same way. What would we call this way of doing things, "paint mode"? We never thought of it as a "mode" at the time because there wasn't any other mode.
In theory in immediate mode widget library you can have global state that defines which part of screen has to be repainted if any, but there is not that much of performance benefit because you still end up doing most of the processing for each frame and when there is any damaged region you still in effect redraw everything and clip it to the region.
Well, developers who write their own GUI toolkits (what the article is about), probably know retained vs immediate mode. Those are two very basic distinctions when it comes to GUIs.
Retained mode is the average GUI toolkit you probably know (Swing, GTK, Cocoa, etc). Immediate mode is where the GUI is draw directly frame by frame by the application.
In retained mode you deal with buttons, windows, panels, checkbox objects, textboxes, etc, arranged in a tree-like structure.
In immediate mode you deal with painting commands (of course you usually will abstract them to a higher level too). You want a button? You draw a square and the label in it, and depending on the user input (e.g. clicking at the moment or not), you might draw it depressed.
Not quite. Certainly Cocoa, and probably the others as well, are mixed mode.
An NSView has a drawRect:: method that draws that particular view in "immediate mode", with drawing commands of the underlying procedural graphics API (essentially Quartz/CoreGraphics).
If you want pure immediate mode, you create a single NSView and just draw everything.
If you want pure retained mode, you exclusively use predefined widgets, which hide the "immediate mode" drawing that they do.
Or you mix and match as you see fit.
Yes, but that's the case for almost all retained mode GUI toolkits (they provide something akin to that), and it's used as a last resort (when you need a custom look or to implement your own widget, or complicated UI for something like plotting or a piano keyboard, etc).
Regular use of the APIs in Cocoa (and what all books advice) is retained mode style, using pre-made widgets with the Interface Builder and so on. Not "drawRect".
Usage appears to have shifted that way. However, that wasn't the case not so long ago and the fact that a facility is no longer being used (as much) does not mean it's not there. If you need it, it's right there to use as you wish.
Just because you tend not to open the roof doesn't mean your convertible is no longer a convertible, as long as you still can open the roof.
And yes, I did mention that other toolkits probably work the same way.
I recall having discussed this with you here again.
I'd say it was the case for as long as I'm on IT (so, as long as mid-nineties or so). Heck, even back to Delphi and co.
>Just because you tend not to open the roof doesn't mean your convertible is no longer a convertible, as long as you still can open the roof.
Well, if 99% of the car has been designed for one use (A), but has the option (that few use) to do B, doesn't mean the car's a B.
So, this is more like "just because the car has a lot of storage space in the trunk, doesn't mean it's a truck".
Late 80s for me. What you saw depends very much on which part of "IT" you were in.
When doing forms automation, maybe you only ever saw the widgets. And that's OK. However, forms/widgets are not all of IT and all of UI work. So when doing graphics, DTP, CAD/CAM, document processing etc.: not so much.
>if 99% of the car has been designed for one use
But it hasn't. Not 99% and not one use. These frameworks are flexible and were explicitly designed to be flexible. drawRect:: and handling mouse events is a fundamental feature of the view, and widely used. The fact that you have convenience widget libraries that encapsulate its use (they ALL use it), doesn't mean it's not there.
Just because you use your truck like a car doesn't mean it's not still a truck.
Yeah, haven't done graphics, DTP, CAD/CAM, etc, and I guess immediate mode or mixed mode is more prevalent there.
But I'd say the vast majority of GUIs (whether enterprise or desktop/business apps etc) are forms and regular widgets based. If they need some custom part (e.g. a plot or a custom drawn control) they implement it by drawing it, but still use it in retained mode, in the classic widget tree, embedded in some window, layout widget, etc.
Immediate mode is almost non-existent in enterprise GUI work for example -- except niches like CAD apps for factories or some special case (a custom widget drawn but embedded in an otherwise retained mode UI).
>These frameworks are flexible and were explicitly designed to be flexible.
Yes, they have escape hatches to draw your own widgets when needed.
But their main idea wasn't "immediate mode", but "we provide these ready-made widgets", and they're judged by how many and good those are, not by the ability to drop in drawRect or such.
In the same way, that OpenGL and DirectX is 99% immediate mode, regular GUI toolkits is 99% retained mode (on the API level programmers are meant to use, and mainly use).
Almost any book on GUI programming on Cocoa, GTK, QT, etc I've read, primarily focuses on the available widgets and retained mode use, and any "drawRect" equivalent is an aside about what you can achieve if needed.
(a) not convinced that's true
(b) not relevant
> But their main idea wasn't "immediate mode",
No, as I've written, their main idea was to provide mixed mode. On top of which you can provide a widget set. Something like GKS or Phigs was true retained mode graphics, and toolkits like AppKit were reactions to the limitations of those types of mechanisms.
There is a reason the Mac had QuickDraw and NeXT had DPS and not GKS or Phigs. Well, many reasons, but one was that the limitations of pure retained mode graphics systems were well known.
> they have escape hatches to draw your own widgets when needed.
Yes and no. While you can view them as escape hatches if you're primarily doing widget-only programming, they aren't really escape hatches as the entire system is based on / built on them. Just like in Objective-C, objc_msgSend() is not the escape hatch, it is both the basis for all messaging and it enables the escape hatch, which is -forwardInvocation:. Similarly -drawRect:: and responder handling: they are the basis on which everything else is built. Since they are exposed, that means you can build your own views/widgets as peers to the ones provided.
> Almost any book on GUI programming on Cocoa ...primarily focuses on the available widgets
The books I learned AppKit with focused on creating a custom view, drawing with DPS and pswrap, handling mouse events and the responder chain.
Browsing the Dear ImGui project, it looks pretty cool.
https://github.com/ocornut/imgui/wiki
From the hip, I'd say one big difference is the API for modals is blocking (waiting for user input), vs callback. That's pretty cool.
I don't yet know how non-modal is handled. Edit: Ah, it looks like return values are also wired up directly, vs the indirection of an event model or listeners. Clever.
--
At this time, I don't think comparisons (in this thread) to OpenGL's immediate ("interactive") vs retained ("batched") modes are relevant.
For a game, which is already quite busy even when "idle" (and there are probably other things going on in each frame which take far more CPU and GPU to render), it won't be much of a concern; but with an application where most of the time is spent waiting for user input, spinning the CPU to re-render the GUI every frame is very energy-inefficient.
Thus the Performance Argument against any IMGUI is pretty pointless.
So while Electron will lose in memory/disk used, IMGUI will lose in CPU usage (battery life) once the software is complex enough.
Electron (and nearly all retained mode GUIs) will execute 10x more code trying not to execute the rendering code. Just skipping all that checking and jumping to the rendering code is arguably a win in many many cases.
To put some empirical data on it, game devs use ImGUIs because they keep up with perf where as that 10x overhead of retained mode guis creating gui objects, deleting gui objects, marshaling data into and out of gui objects, adding and removing event handlers and callbacks, and then trying to write a bunch more code to optimize all of that never leads to a GUI that is performant enough for the GUIs needed for game dev.
I think if you check actual apps, especially phone apps, you'll find it's not so clear which one wins. Phone apps almost always move in relation to the users fingers meaning the screen is being re-rendered. Usually new GUI elements are moving on and off the top/bottom of the screen. The overhead of creating/deleting/managing those objects is arguably greater than just re-rendering. The screen is already going to be re-rendered. At best you're getting a few cached textures from your retained mode gui all at the expensive of managing GUI objects.
For desktop apps it really depends on the app. It's got nothing to do with the complexity of the UI and everything to do with what the app does. A text editor might be a win for a retained mode GUI. Photoshop/Maya it's less clear. For example Unity is ImGUI and has very complex UI. Much of the UI has to respond in realtime to the state of the user's document that is changing constantly.
I'm not really familiar with the mobile platforms, but given their reliance on battery power, I hope(!) that scrolling is done mostly in hardware and actual re-rendering happens only for areas beyond the scrolling boundary. Of course you will need to re-render when the actual content changes, but not with just scrolling.
To put some empirical data on it, game devs use ImGUIs because they keep up with perf where as that 10x overhead of retained mode guis creating gui objects, deleting gui objects, marshaling data into and out of gui objects, adding and removing event handlers and callbacks, and then trying to write a bunch more code to optimize all of that never leads to a GUI that is performant enough for the GUIs needed for game dev.
That sounds more like an implementation/application problem --- overuse of OOP or an application where too much is constantly changing --- rather than an inherent inefficiency of retained mode. The Windows native GUI, since Win16 in Windows 1.0, was basically designed for retained mode. Immediate mode is more common in games because it's more suited to that application.
How do you figure? Rendering a single Truetype letter will have dozens of conditional branches, use complex floating point math, and touch tens of kilobytes of cache. And there are many thousands of them in the GUI. Even if you have rendered cache, there is still lots of memory pressure copying data back and forth.
Compared to that, retained mode GUIs should be much more efficient. Even 50 checks are nothing compared to ability to void rendering one line.
Note this is all predicated on screen being mostly unchanging. For example, as I am typing this comment, everything is stationary except for a single line where I am typing.
> The screen is already going to be re-rendered.
This is the key! As you said, if you have to re-render whole screen anyway, then retained mode GUIs will have to maintain draw state and re-render every element every time anyway -- a strictly worse performance.
But most of the regular apps, like editors and chat clients and web browsers, do not re-render whole screen every time. They only do one thing at a time.
For an empirical evidence, look at WinAPI design: this is a retained mode GUI, with lots of effort dedicated to figuring out invalidated rectangles and which controls need to be rendered. This was the only way to make a GUI which is performant enough.
There's best case for Retained mode GUIs and best case for ImGUIs. ImGUIs excel when lots is changing. In a scrolling mobile app the entire page is being re-rendered constantly as the user moves the page up and down. GUI widgets get created and deleted, data gets marshalled in and out, various algorithms are applied to try to minimize re-creating stuff, all code that doesn't need to be executed in ImGUI mode code.
There are tons of cases where ImGUIs win in perf. In fact the reason they are so popular with for game dev is exactly because they vastly out perform retained mode guis. There's a reason there are 100s of books and 1000s of blog posts and articles trying to get retained mode guis to run smoothly. They almost always fail without lots and lots of specialized client side code to try to minimize GUI widget object creation/deletion, reuse objects, minimize state changes, etc, all that code disappears in an ImGUI and they run at 60fps where as all the retained mode guis struggle to get even 20fps consistently as they hiccup and sputter.
Smalltalk/Self GUIs:
AFAIK, the idea (or at least terminology) of MVC comes from the Smalltalk80 GUI. MVC seems to be intrinsically retained-mode: the "view objects" (buttons, forms, etc.) are separate to the "model objects".
Self was inspired by Smalltalk, and its Morphic UI seems to be immediate-mode: the UI objects often are the model objects. This idea is more explicit in the "Naked Objects" approach. Squeak Smalltalk uses Morphic, although its descendents (e.g. Pharo and Newspeak) seem to have replaced it (I'm not sure of their paradigm though)
Denotative, Continuous-Time Programming (DCTP):
This was originally called "Functional Reactive Programming" ( http://conal.net/papers/frp.html ), but that term is now mainly applied to discrete, event-driven systems like Elm and React, which can make some articles/papers a bit confusing to read ;)
In this approach, anything which varies over time (e.g. user input, display widgets, application state, etc.) becomes a function taking the current time as a parameter, called a "behaviour". These behaviours can be composed together in various ways to create an application's overall behaviour, still parameterised by time. The "main loop" just samples this behaviour at an appropriate rate, e.g. a fixed frame rate, or varying based on system load, how much activity is happening, etc.
This is very much IMGUI, but has some nice advantages over the fixed ticking of a traditional main loop. For example, different components of a behaviour can be sampled at different rates, e.g. rapid sampling for smooth animations, coarse sampling for expensive calculations. Interpolation behaviours can remove jerkiness (e.g. a smooth animation based on an expensive simulation).
Pharo is still using Morphic, though there is a new graphics layer called Bloc (vector and constraints based) that has been under development to replace it.
For example, part of the update function might look like this:
if (Button("Do That Thing"))
doThatThing();
An immediate mode GUI is basically the opposite of a typical Model-View-Controller GUI because it deliberately mixes the Model, View, and Controller together into one function. Just one function draws the GUI based on whatever state it's in and whatever the model is. The same function gets reused in a different context to update the GUI state or handle callbacks.
Also ImGUIs are not multiple pass. That's in implementation detail. Unity's ImGUI is multi-pass. One of the most popular ImGUIs (Dear ImGUI) is single pass
For my first example, updating controls and overall layout in a WPF application feels much more lightweight than in WinForms because the renderer just shows the result at the next monitor refresh instead of trying to update the layout in realtime or using hacks that disable redraw until you do everything.
For the second example, I've used SciChart to display trading data and it again was much more responsive because it replaced a control-based rendering pipeline with just an immediate-mode canvas tgat would render the selected data window once per monitor refresh rate.
And I think overall once you hit the performance barrier where you can't just fill a panel with controls and let them render and instead have to make a custom draw routine, the immediate-mode rendering adds this extra smoothness to the UI and gives you a bit more performance headroom.
Terrible performance, doesn’t scale to UIs more complex than a form (... like the ones many games tend to have)
It got to the point where there was a website dedicated to track when we’d get the current non-immediate UI, and Unity actually hired the developer of the leading non-immediate mode UI library in the asset store to work on their own offering.
The reasons they switched the in game system are many but they have nothing to do with performance.
... but IMGUI is on the way out there too: https://docs.unity3d.com/Manual/UIElements.html
The old system would stack up draw calls like nobody's bussiness, doubling drawcalls if you used the built in layout tools. OnGUI being called period would be expensive, even when you had properly implemented your rendering logic, so you’d have to disable the entire game object when you didn’t want to draw.
The docs for mobile literally used to say not to use the OnGUI stuff during gameplay.
How do screen readers work anyway?
Maybe you're right and you can send the text to be read directly to the screen reader. Not sure how well that works when your GUI updates every 20ms. You don't want to keep resetting the reader.
Anyway, I believe Qt is screen reader compatible, so it must access the relevant API somehow. I'd be surprised if it was the exclusivity of the OS vendor.
One of the new features of 7.0 was a library for producing text mode GUIs with dialog boxes and the like. Because the language didn't support anything resembling a callback, everything had to be written in what was essentially "immediate mode". Lots of control flow that would normally be down in the library stayed up in user code. The net result aligned well with the description in this article: "...much shorter but more intensive, every thing is packed in a single function."
This approach also presented lots of opportunities for errors caused by failing to correctly code something that would be default behavior in a normal library.
Never seen anything else quite like it, aside from this. (Which, quite frankly, I'm happy about.)
Immediate vs deferred, is just another paradigm that goes back and forth in trends like keyboard input/buffering, deffered/one pass lighting, monoliths vs modules, State vs events, flat vs 3D
To be honest, given that web render output is a tree instead of flat "render to screen" stuff, writing a nice immediate mode GUI is not obvious.
If you're... very clever you could take React and the State stuff, at least conceptually, and get what you want here.
imgui's model is "you draw onto pixels". You could create another mental model which is "you draw onto a tree." so instead of saying "drawText(x, y, text)" you would do something like "drawText(domLocation, text)"
Granted, this model gets a bit messier, and _some_ might say defeats the purpose of immediate mode. But I think you could still get conciseness benefits on this.
There's an introduction/tutorial here - https://github.com/ajnsit/concur-documentation/blob/master/R...
Sneathlane is a little different because it outputs to canvas and has a rather complex Widget type to handle focus and to allow different outputs to work together, but the Monad instance has the same behavior of waiting for widget completion. Imperative leans more on the DOM, it has its own VDOM implementation where yours looks to use React's but otherwise the libraries look very similar.
I'm a web developer, I've worked with everything from FRP to React, I think this approach is the best. I hope Concur will help to popularize it! I'm writing this on my commute but will be looking into this further today, to see if there's anything in Imperative that might enrich Concur or vice versa.
In this post the author seems to go further than two buttons, but I'm skeptical. I wish there was a video showing how it works in dynamics - inputs clicked, texts entered, scrollbars scrolled, dialogs dragged around. Or source code available to build and try locally.
From the code in the post I suppose the majority of UI is dead - can not be interacted with.
I think IMGUI can only be convenient for very simple cases.
In real frameworks the lowest level is immediate mode (see WM_PAINT in WinAPI), but real UI needs quickly evolve them into the traditional RMGUI.
BTW, this thinking is my source of doubts about React.
One of the most complex apps out there, Unity, is entirely ImGUI based for the app itself and often has 1000s of controls visible. Going though the screenshots of typical ImGUI usage most of them are on the complex side.
https://cloud.githubusercontent.com/assets/8225057/20628927/...
https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v16...
https://user-images.githubusercontent.com/16171743/50757906-...
https://user-images.githubusercontent.com/8225057/51042368-6...
https://user-images.githubusercontent.com/870320/51073482-ba...
See https://github.com/ocornut/imgui#Gallery for more