Show HN: Mixing WebGL and HTML, Both Rendered Out in React
subdued-health.surge.sh
subdued-health.surge.sh
The main thing I knew I wanted was a declarative specification of the objects in our scene. I knew it would be a nightmare to fiddle around with manually adding and removing bits of the scenegraph in response to input.
Ended up just doing diffs on the part of the Redux state that mapped to the scene, and automatically discarding and re-running the mesh construction code for the parts of the state that changed.
It actually worked well enough and didn't cause any problems I noticed (in terms of convenience of use and performance), but I wouldn't want to re-write it for each project that could use it.
One thing I do wonder about with this though is whether there's really a benefit to using the tag language to specify three.js objects. For folks already familiar with the three.js library it's going to be a nuisance to use the tag language instead, and it doesn't appear to me to be more concise or convenient in any way. But maybe I just feel that way because I've been writing js to construct three.js objects for years :)
The benefit of using tags (jsx) is that objects become fully managed. The most clear indication to me is still effects. https://codesandbox.io/embed/mo0xrqrj79 You can drop that in and it works. Make it conditional and once it gets unmounted the render loop goes back to normal. Almost magic, but then again, react for the dom had the same benefits.
> The benefit of using tags (jsx) is that objects become fully managed
I'm not sure I follow here. I assume that by 'become fully managed' you mean that React will reconstruct as necessary based on what is declared in the jsx?
So maybe that is just the nature of writing React renderers specifically, the only option for getting your objects managed is to map them to jsx?
> The most clear indication to me is still effects.
This does look much nicer than other examples I've seen of using effects. I have only limited experience with them personally. They weren't needed for our tool. Incidentally I am probably setting up some SSAO on another project today though so I'll be likely be digging more into them later.
{condition && <div>hello</div>}
We did this manually before, which usually meant watching out for objects to remove themselves, they'll have to communicate up their hierarchy to inform parents and so on. This wasn't straight forward in three, so there was a lot of prototype hacking and terrible things like that, i'm so glad this will be all gone.
http://symbolflux.com/projects
I've been working on a map editor of sorts. Please take a look!
Three.js has had something called the CSS3DRenderer for like 5+ years now. It uses CSS 3D transformation effects to proportionally scale/distort HTML content to what the Three.js viewport is doing. It does most of the heavy lifting for you in terms of matrix math, but you still need to write a lot of glue code if you want a unified feel. I worked with it a lot in 2014, and it felt like alien technology at the time. Ultimately I settled on purely immediate mode though, because you can't apply custom post-process effects to DOM content beyond whatever CSS effects offer.
Famo.us as it originally existed launched in 2015 and was all about mixing WebGL/HTML content, but they pivoted shortly after launch. Some interesting developments came out of that, however:
Their VP of Product, Zach Brown, ended up founding Haiku in 2016, and their first product Animator[0] just officially launched for Mac. I view it as a more capable replacement for Adobe Edge, which was a fantastic little tool for animating HTML content. How flexible Animator actually is in terms of mixing GL/HTML content, I'm not sure. It'd probably need to take the modern Adobe Flash approach and decouple the tool output from the underlying client runtime to succeed at that.
Following the famo.us pivot, community members founded a project called Infamous with the goal of keeping the mixed content dream alive. While most people fell away, one person (Joe Pea) did stay the course and has spent the past few years since then developing Infamous[1], which is basically an entire framework built off the premise of mixing GL and HTML content.
I'm not sure if it's ready for prime time yet, but it sure looks good and I'm extremely impressed with Joe's perseverance. Just looking at the commit history messages, you can kind of get a feel for all the little issues that mixing GL and HTML content creates.
It put my laptop on Takeoff mode (fans full speed!) with Firefox.
None of the demos work on Firefox.
Non sequitur.
Your computer is doing more processing. There's WebGL, which will engage your graphics card. If it is on your CPU, the CPU will consume more power. Depending on your laptop, you could have a discrete graphics card, which will be switched to and consume more power. There are animations, etc.
It is not only because of React. In fact, by avoiding unnecessary DOM changes, it's probably an advantage (although this would have to be measured).
Unless you are telling us that you were playing fortnite or something and your fans didn't spin up, but this thing caused them to spin up.
I run multiple VMs and do 3d work (in fact, mostly direct opengl stuff) on my laptop and it hardly ever goes to takeoff mode with just some simple rendering.
It's a work-in-progress demo of react-three-fiber. There's a good explanation of what it is on the Github page - https://github.com/drcmda/react-three-fiber
Firefox 65.0.1 (64bit), Linux (5.0.0).
There are a couple warnings about a particular extension:
> warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
The rest just appears to be successful parsing messages.
(But it worked really well in Konqueror & Chromium, so that's something.)
Safari and Chrome both had to work around the same thing, so it seems like there's some contention around who is to blame here.
Either way, three.js is (partially) broken for Firefox users.
I used React for VR/AR, to interweave three.js (objects, assorted cameras and shaders) with assorted DOM trees (left and right eyes, foreground (pixel aligned) and background (high-bandwidth - video and such)). This looks helpful.
Most current VR/AR exploration uses Unity (Windows) on vendor platforms (SteamVR (Windows), ARKit, etc) and Vive/Rift/WMR/phone hardware.
Three.js is focused on WebVR. WebVR has been Windows-only. And very not transparent about status (very "aspirational" reporting). I fuzzily recall some year-ago VR work in a three.js dev branch. You might also look at A-Frame[1] and AR.js[2]. But this is not an area I keep track of.
The three.js WebVR folks do aspire to "automatically". Historically, it very very hasn't been. Add more "very"s.
Now, big picture, I'd be unsurprised by the transition to AR/VR being shockingly rapid. And unexpectedly soon. So there might now be some path that "just works". And it could be worth starting to track, so if there's a new niche you might like, you're aware of it.
But historically... VR, noun: A field covered in hirsute yaks. And the trampled bodies of people's years, expended shaving them in hope of wonderful fruit, while it was still unripe.
[1] https://aframe.io/ [2] https://github.com/jeromeetienne/AR.js/blob/master/README.md
Map Editing
https://www.youtube.com/watch?v=CE2Apyf-KOE
Angled View of the Scene