Old Mozilla / Firefox with XUL / XPCOM[1] was probably about as close as you get. If you were willing to use the low-level interfaces, you could change just about anything as far as I can tell. But that led to security issues and issues with maintaining comparability when there was a need to change those low-level interfaces. So Firefox largely moved away from XPCOM and blocked extensions from calling XPCOM interfaces.
That said, AFAIK Pale Moon still allows use of XPCOM. And I think Seamonkey may be the closest remaining browser to the Mozilla of old. I believe you can still use all of the XUL/XPCOM stuff there, but don't quote me on that.