> It would be good if there was some fallback mode where Firefox would render UI elements using HTML itself, without relying on UI toolkits.

This isn't quite a meaningful assertion. The way it works in Firefox is that a UI developer specifies in a xul or html document that there should be a button:

    <button id="myButton" label="Click me!"/>

Then Gecko applies several CSS rules for button elements. One tells it what binding to use (https://dxr.mozilla.org/mozilla-central/source/toolkit/conte...) (a binding is an means of abstraction that packages up behavior and document structure). The other tells it what appearance to use (https://dxr.mozilla.org/mozilla-central/source/toolkit/theme...).

Note in particular the `-moz-appearance: button;`. This tells the rendering engine to figure out how big the button is, ask the OS to draw a button of that dimension, and then composite the OS button on top of the button node. I'm not an expert but a quick glance at the code points me to https://dxr.mozilla.org/mozilla-central/source/widget/gtk/ns..., which seems to be the main entry point for Gecko to ask for a widget to be rendered.

As you can see, Firefox does not rely on a toolkit to tell it how buttons _work_, only how they _look_. You could supply a stylesheet that overrides the -moz-appearance rules and you'll get buttons that are rendered purely by Firefox. (You might need to supply some more style rules so that they actually look like buttons.)

If you want to compile without GTK, that's a little more complicated. Firefox uses the toolkit for things like the message loop, mouse and keyboard event processing, etc. You could do all of that in straight Xlib, but who would want to? Certainly nobody has wanted to enough that they've been maintaining an Xlib port all of these years.