In the end, HTML is just a standard for defining view element hierarchies; CSS is just a standard for specifying declarative layout constraints on those element hierarchies; and JavaScript is (supposed to be) just a standard for specifying how interactions with those views are bound to your view-controller. (The view-controller in a browser originally being an NPAPI <object> like a Java or ActiveX applet.)
None of these standard formats has any particular tie to the way web browsers use them. The standards, of course, go beyond syntax, and also specify the HTML elements, CSS properties, and JS DOM that define "the web" as a platform. But it's perfectly possible to use HTML, CSS, and JS syntax—with standard parsers!—to power your own windowing toolkit.
Define your windowing toolkit's view hierarchy format as "HTML syntax, but with these valid elements" (where they map to the controls in the toolkit.) Define your windowing toolkit's layout constraint properties format as "CSS but with these valid properties", where these map to the properties the toolkit exposes on the controls. And finally, drop-in a JS engine, but don't tell it to expose any APIs to the view to control things from Javascript; rather just mirror the API of the view-controller attached to the view as a set of opaque callback functions that can be hooked up to the elements of the view with addEventListener.
XUL or XAML could have just been this. ePub is basically just this (though in some implementations a webview is used.)
And, as a side-benefit, it becomes very easy to, like ePub, define your element and layout-constraint formats as supersets of HTML and CSS, such that people can still use familiar HTML elements and CSS properties for inline rich-text styling inside <label>s et al, while still using your toolkit's controls for everything else.