Render HTML via WebGL
github.com
github.com
It using html2canvas[0] to rasterize the browser rendering into a canvas, upload it to WebGL, apply effects and patch-back events(i.e. mouse clicks) back to the hidden browser dom elements. (On the other side, font hinting and such are not really a problem)
Implementing full UI framework over WebGL has it merits, and hopefully someone will make a useful(and not html+css based) one some day.
Famo.us is the only JavaScript framework that includes an open source 3D layout engine fully integrated with a 3D physics animation engine that can render to DOM, Canvas, or WebGL.
I'm surprised it hasn't been done already (in similar spirit to the Feathers framework for Flash's Stage3D). Now that WebGL is supported in Mobile WebView, maybe you'll see a really good component set released? That would give devs a really slick option for building smooth phone-gap based apps.
If we're going to build a new rendering engine for the web, maybe we could try to create something nicer than HTML and CSS, whose only advantage at the present is the number of developers familiar with it. I know I wouldn't miss them.
le sigh.
Many other UI frameworks have had Flexbox type layouts, a technology W3C has only just recently realized is a good idea for 8+ years (probably much longer).
This all evolved over time into the clusterf-ck we have today, setting us back probably 20+ years in terms of developer productivity.
I was commenting on a hypothetical future for something of its kind, and that I think HTML content is not really the way to go to make shiny WebGL things, IMO.
To be a little more on-topic, yes, as an "apply shader to this semi-static div" it is a pretty cool thing :)
Previous HN thread : https://news.ycombinator.com/item?id=6314961
I'm saying this as I have compared the performance of a simple local web page with a simple view with the same exact functionality rendered with my pet naive unoptimized UI engine and the latter was blazingly fast compared to HTML.
So I guess you are right, we definitely need something less cluttered and more efficient than HTML+CSS+JS.
Once you start doing anything outside of your simple page with a simple view (which was what, incidentally?), your engine starts to accrue features that move it in "bloat" towards a modern web browser.
The link mentions "Hey, this is really useful on mobile"--and I would consider that a failing of the mobile vendors browser teams, not a problem with the web.
This looks really flashy for the sort of static content + basic animations + interactions the author is showing off, but once you start handling further interaction--like, you know, building a virtual dom to render to the webgl texture, all you're doing is reinventing the facilities the browser already gives you, but instead behind one more layer of abstraction.
tl,dr; web developers never learn.
Actually, my pet UI engine was powerful enough so that I've implemented a complete mobile application in it. I'd also tried HTML, but it just wasn't competitive. The native iOS UI was even faster and was miles ahead of HTML in terms of speed.
Also, there is such thing as Qt Quick, where you can basically write in reactive JavaScript, it is at least as powerful as HTML-CSS-JS, and it is still way faster.
The web has started from something that obviously wasn't designed for performance and has come a long way of incremental evolution. Given this, and given that I know how hard it is to make a fast UI engine as powerful as web, the current web performance is actually spectacular. But still, there are better models, and there can be even better models.
So, that's part of the problem, right? For some damn fool reason we keep throwing beginners at frameworks--they should be starting with just bsaic HTML and CSS, and then learning JS, and only then moving on to frameworks. It's so damned confusing in part because people aren't learning the fundamentals, and so are lacking a mental model of how things work and what makes them slow.
I'd also tried HTML, but it just wasn't competitive. The native iOS UI was even faster and was miles ahead of HTML in terms of speed.
Show me some numbers. If you want to talk about performance, then let's be quantitative. I'm also willing to wager that the HTML on the mobile device was full of bits you didn't need.
But still, there are better models, and there can be even better models.
Again, why do you think this is the case? It's worked, with expansion, for over twenty years. That's a staying power few technological approaches have--maybe, there's something to it. This notion of "better models" really frequently boils down to "I have this one specific thing I need done, and anything that doesn't match this one specific thing is superterribad and clearly needs replacing."
If you want the best performance (which honestly probably doesn't matter that much for your use case), write things in native code as you've done. But don't then pretend that that is a valid yardstick to measure an entirely different tech by.
Actually, I've been familiar with HTML+CSS+JS for a long time. The vanilla web is just not very practical for creating full-fledged web apps, especially given that there are so many frameworks that have a lot of things out of the box. Also, the vanilla web misses some features that are essential, such as the reactivity (binding).
Show me some numbers. If you want to talk about performance, then let's be quantitative. I'm also willing to wager that the HTML on the mobile device was full of bits you didn't need.
The last time I tried you wouldn't need numbers, as my pet engine did steady 60 FPS full-screen animations, while the iOS web view pathetically presented something closer to a slideshow, Android did better, but still not smooth enough. And the page loading was visible instead of instantaneous display with the native stuff. Of course, faster devices alleviate the problem.
>But still, there are better models, and there can be even better models.
Again, why do you think this is the case?
Because I've developed my own thing similar to what HTML+CSS+JS do, and I could clearly realize that you can do the same things with a much simpler model that also happens to work faster. Also, there are existing native frameworks that are examples of this.
It's quite clear that the web wasn't originally designed to be used how it's currently used and practical web apps are built with basically hacks over hacks and hack-based frameworks. Obviously, getting rid of all that clutter accumulated by incremental evolution would make a difference.
> Actually, I've been familiar with HTML+CSS+JS for a long time.
I am familiar with HTML, CSS, and JS, but can't make anything except the simplest static website.
My point wasn't about how good my pet engine was (because it wasn't), but about the fact how the overcomplicated model of the HTML-CSS-DOM-JS makes it so hard to optimize web page rendering that even an unoptimized prototype UI engine outperforms the browser.
I am amazed by the people who develop browser engines because they do something almost magic of fitting that monstrous dynamically modified DOM into so little memory and making it so fast.
I conjecture that in a naive unoptimized implementation a web browser would consume gigabytes and took a couple of minutes to load an average page.
Given the sheer number of people familiar with it, I'd say this is a pretty huge advantage.
That's the opposite direction you want to go. At that point, you might as well just use HTML. I think there are benefits to providing a subset of HTML features that you can tailor for super fast performance.
In the end the only difference you'll have compared to the real browser renderer is that you do less computations from CG(GL) to RAM (JS), hence the performance.
You can check this very old project of mine where we tried to link DOM and WebGL Nodes: https://github.com/aout/SAGE
edit: just saw the repo wasn't entirely up to date. Done now.
An example of that is drawing a DIV and applying a CSS 3D Transform to it. By doing so you'll modify it's display on the screen so you'll need to recompute it's actual bounding box. If you compute the BBox on the GPU and never access it anywhere else, everything is fine but as soon as you have to use it in JS you're screwed because you'll have to transfer data from your GPU to your RAM. Doing so at 60fps is very expensive, then you'll take a performance hit from JavaScript being usually slower than C/C++.
That's not the only problem you'll have, text rendering is probably the biggest one. As you might know, textures are rendered using memory friendly algorithms. To ensure maximum readability and quality you'll have to (1) use a lot of memory and / or (2) apply some advanced filtering so your texture doesn't lose quality. Those techniques are quite hard to implement if you're not a 3D programmer (just check what happens if you try to rotate text in CSS and select it) but more importantly you'll find yourself reinventing the wheel.
Even then, the main problem is that the internet is still mostly text, and if you can't implement or access font hinting, your text will be blurry and ugly on any display that isn't ultra-high DPI. You can't really render text to a texture and then manipulate that texture with sub-pixel precision. It must be done the other way around: text rendering must be done last, and with the help of a hinting library so it will look sharp and readable.
It rasterizes the DOM.
I think that's important enough to warrant going through the paces and implementing this. Don't listen to the naysayers. Reinvent wheels. The wheels we currently have largely suck.
I'm curious if there would be any conflict with usage of Three.js on the same site? Could they end up clobbering each other's textures? (I've not gotten into webgl's texture memory management, so that question is a stab in the dark.)
On the topic of Microsoft and the web, ActiveX gets a lot of bad rep (and for good reason), though it seems like they were trying something akin to web components back then. It worked better in actual applications than the web though.
Reference - https://news.ycombinator.com/item?id=9029159
I've experimented with canvas in the past to see if it could improve the performance of, say, a terminal that updates very quickly but it ultimately ended up being just as slow as the regular DOM because of all the extra code that had to be running in order to keep the hidden buffer in sync with what was displayed to the user.
I'd be curious to know how the performance of WebGL compares to the (accelerated) canvas.
Also, css shaders.
I wonder what exactly the use of this is, beyond Compiz for web pages. What's next, jiggly divs? :P
moron4hire made a good point about letting devs pick the layout engine, but I am not sure we want to ignore all the nice features browsers give us here.
http://www.pouet.net/prodlist.php?platform[]=JavaScript&page...
Obviously, this can be done at the browser engine level.
I have also tried to do this in my pet UI engine, but found out that it was tricky to figure out caching rules to avoid memory blowup and performance degradation for all possible cases.
This approach has proven itself very well in iOS, in fact, it was the absolute best mobile animation technology for a long time. Actually, the older versions of iOS used software rasterization with hardware accelerated bitmap caching and it worked smoothly on the original iPhone.
one might be a crazy heavy SPA or javascript webpage that lags and show it through HTMLGL.
this really could go somewhere, the idea is solid but it might take some work to actually deliver, a constant fps web.