Opening up cyberspace with Matrix and WebVR | Hacker News Reader

Opening up cyberspace with Matrix and WebVR

matrix.org

76 pointsvanburen34 comments

I'm huge on VR -- especially WebVR -- but this isn't what comes to mind when I think of the metaverse opuses in the post. Telepresent chat rooms were never an important part of these stories. That's not where the magic is.

My take is that the metaverse is simply a software system so deep that it becomes part of everyday reality. And we've already built this, but our human-computer interface technology (mostly the software) sucks too much for us to see it. In the meantime we call it the web.

This looks like really cool tech, but I don't think the path to the matrix involves a new protocol, or a new data layer. I think it will come about by extruding our pre-existing software epic -- the web -- into our virtual and augmented environments, in the same way the iPhone compressed the software and ideas that already existed into our pockets.

The problem is that the web itself just doesn't have a standard realtime component. (other than Matrix, or possibly XMPP or Solid). Sure you can provide better HCI technology to expose the websites and webservices of today to users via VR/AR... but they end up just being disconnected silos; the equivalent of different tabs in a browser.

The thesis here is that you need some kind of fabric to weave these apps together if you want them to feel remotely immersive in AR/VR. Rather than each app being its own island, surely you want the ability to move your avatar between them, and collaborate and communicate between services and apps, and have some kind of common virtual physical metaphor that you can literally build on as a platform - whether that's with bots or services or whatever. This is the thought experiment we're doing with the VR side of Matrix.

Are Websockets and WebRTC not realtime enough? They need to bake more, but they're literally standards.

I wholeheartedly agree that probably the biggest problem that needs solving is that of siloed experiences. But this is also something the web solved long ago with URLs and files on the front, and webby APIs on the back.

If we invent another protocol to solve this problem, are we not simply creating new silos?

Websockets and WebRTC really are not enough at all, and this is a very widely held misapprehension. Websockets (and their HTTP/2 equivalent) are just a dumb pipe for shifting data - just like a TCP socket, but tunnelled over HTTP. Meanwhile WebRTC defines how to encode and transport real-time media streams over the 'net... but deliberately avoids specifying how to discover or signal the existence of those streams in the first place. In analog terms WebRTC is like a phone line without a phone exchange.

I agree that the open Web avoids silos through URLs and ad hoc HTTP APIs. But this is insufficient for a hypothetical VR universe of immersive services: you don't want the only way for a user to be able to navigate between services to be to 'click a link'. And it's arguably a failure that almost every single web service ends up publishing its data via an incompatible ad hoc HTTP APIs - for instance, given Facebook, Slack, HipChat and Skype all provide supersets of precisely the same basic functionality, how come each has its own proprietary custom HTTP API to do the same thing? Wouldn't it be better if there was a standard open data fabric into which they exposed their data?

The idea here is that VR will suck if it's just a bunch of disconnected apps, and instead you need decentralised primitives for sharing Identity, Avatar properties (e.g. position & orientation), shared VoIP, shared Video, even shared world geometry/assets/physics in order to stitch it together into a coherent experience; both for users or developers.

The web was successful because HTML is so simple, forgiving and flexible - and you end up with essentially the minimum successful viable hypertext system. One could probably limp along with a VR Web which is just a bunch of webapps which happen to have chunks of WebVR UI. But it could be so much more, while still being flexible and forgiving and lightweight. At least this is Matrix's hope :)

totally off topic, but misapprehension is such an oddball word. It doesn't mean what I would have thought it meant. Totally threw me for a loop and I've added it to my list of words that I think people shouldn't use because there are better more common words to use to express the same idea. Thanks!

Apologies for nitpicking, but I find it rather important: Websockets are not tunneled over HTTP, they are only negotiated over HTTP. Once you are connected, there is no overhead of sending HTTP headers back and forth.

I want to find this exciting but I can't help feel a little jaded when I hear Matrix's PR spin this as "the missing link" when we've already have countless attempts at creating 3D worlds like this but which all have since faded into obscurity. Heck, I even wrote my own "3D website" in the 90s using a long defunct web markup language called VRML.

It will be interesting if/when we finally get there but I don't expect this will be the breakthrough product. Though personally I think the real evolution of this tech will be more along the lines of augmenting reality with web tech rather than emulating virtual reality to behave like our physical world.

Everyone's VR PR posits that they're unique and precious and indispensable. Unicorns in goggles.

I'm not sure this is entirely fair, given the blog post clearly tries to mention the other folks who are working in this space. Specifically for the "missing communications/data layer" for VR, the only other options I personally know of are High Fidelity (decentralised, but not yet trying for an open standard or to use the open web), OpenCroquet/Cobalt (seems fairly dead since 2010), OpenSimulator (not decentralised, afaik), and there have been some talk of JanusVR + IPFS hybrids.

Empirically, there certainly is a missing layer here, and it doesn't seem unreasonable to propose Matrix as a possible solution :)

In terms of "this will happen organically, rather than as a whole new virtual universe appearing out of nowhere"... we see AR and VR as entirely the same problem from a comms/data interoperability perspective. Both of them need somewhere to store the data. The choice is simply whether you store it in a decentralised DB like Matrix, IPFS, a blockchain(!) or whatever... or if you store it in a silo like a typical VRML or WebVR app, or something like Google/Facebook.

I don't blame everyone for being cynical though - this is the holy grail after all :D

WebVR concerns me because of the garbage collector and the stop-the-world behaviors you see in browsers. VR is touchy. If you drop frames, you don't notice it just with your eyes, you notice it with your stomach. It feels really, really bad. Getting adequate perf out of a browser for this stuff seems really implausible.

WebAssembly is coming

that's what we thought, but you get a solid 60fps on a typical laptop in a 4 way webrtc video conference, or when showing 4K 360 degree video (plus a video call). My Samsung S5 manages about 30fps in Cardboard mode, which is adequate for Cardboard, and on a proper gaming rig it works flawlessly with a Vive or Rift. Seriously: try the demo at https://matrix.org/vrdemo in browser and see!

The only perf problems we've had are when using VP8 or VP9 for WebRTC video, whereas H.264 (which the demo prefers) correctly decodes efficiently in hardware... and on Firefox where unfortunately https://bugzilla.mozilla.org/show_bug.cgi?id=1246410 slows down the 360 video significantly (to around 20fps for 4K video on the gaming rig) relative to Chromium which manages a solid 30fps (which is the source framerate).

The only motion sickness we've seen has been when editing the skiing videos - the worst thing in the world is if the cameraman wobbles the camera whilst getting up from the floor but your head doesn't move...

btw, hit 'i' in the demo to see the stats. we currently cap the framerate at 60fps.

Video is one thing--you can offload a lot more of that out of JS-land, yeah? What worries the hell out of me is the rest of WebVR. I mean--we see frame drop in Canvas on midrange PCs that's really hard to erase (sincerely, My Last Two Days Of Free Time). The prospect of that sort of mess in VR has me worried. I have a strong stomach so I may play with it once my replacement Vive gets here, but...man. This stuff is so razor-thin on the margins that it really worries me and will continue to so long as we're stuck with stop-the-world GC (and to be clear, of course you can write GC-friendly code...but it's a lot harder in JavaScript, IME, than in Java) anywhere in this stack.

I'd expect 2D canvas to have worse performance than WebGL. I don't think that's a fair comparison for WebVR.

Try out https://aframe.io/a-painter/ when you get your Vive in Firefox Nightly and try out the performance. It's native like for me.

Another thing that seemed really implausible not long ago: that Javascript could be used to build an application. At all.

Many people are already getting 90 frames per second out of the browser, so I don't think it's even a matter of plausibility at this point.

> Another thing that seemed really implausible not long ago: that Javascript could be used to build an application. At all.

DHTML was introduced in 1997. Do you define this as "not long ago"?

"DHTML" was a term long dead while people still thought you couldn't make apps in JS. I doubt he was referring to snow scripts.

So... what I'm reading is that a whole new attack vector for angry teenagers will be to mess up your display in a way designed to make you puke?

Thrilling.

Hey, we already do it at my university. The Vive was a huge step up from the Rift DK2 (which is hardly a fair comparison, but that's what we had) in terms of user rejection due to sickness, but today we still have people pranking each other:

gently tap the tall tripods we have the lighthouses on and it produces a barely perceptible rocking for the VR user, mostly perceptible in their stomach.

It takes no more than 30s to go from perfectly fine to crippling nausea for most people in my lab.

Friends who work in this space treat this as a fireable offense. For the uninitiated, do not under any circumstances fuck with somebody in a headset--the feeling it causes can wreck an entire day. I have a strong stomach--and in VR, games that move independent of the camera don't bother me unless it's prolonged or super sudden--but I was useless for six hours after somebody ran into the lighthouse stand really hard (and caught it, but not before it swung about 30 degrees). I dropped to my knees and had to tear the headset off my head. It ranks up there with the worst experiences of my life.

I think OP meant virtually. Tapping on lighthouses is just like physically hitting them. Virtual attacks have bigger implications.

pretty sure angry teenagers can mess up your display in ways to make you puke without using WebVR :D

Check out the room scale Vive/Rift demos at https://github.com/aframevr/aframe. They run on Firefox Nightly, 90FPS, low latency. The Web has a bad stigma associated with its 2D pieces, but it's definitely not implausible.

http://www.geektime.com/2017/04/04/matrix-org-brings-cross-p... provides another take on this, fwiw, highlighting that the battle for an open VR web has to happen right now if we're to avoid a closed and fragmented future like we have today with instant messaging.

My immediate thought was Cobalt/Croquet. It's nice to see that they mention it, but only as "an ambitious project"; then go on to propose their own (part of) an ambitious project ;)

There are some really compelling things to like about Cobalt; not least the fact it incorporates a load of existing protocols (XMPP for chat, VNC for accessing "traditional" applications, etc.).

Unfortunately, its use of Smalltalk (which probably helped it reach so high so quickly) seems to be holding it back from more contributors and adoption. There are Javascript ports which have seen some limited usage, but they appear to be more constrained. Perhaps bootstrapping Squeak on WebAssembly would help, and maybe offering a layer of JS scriptability on top?

Whilst I wish Matrix the best of luck with their efforts, it would be equally great to see their protocols/ecosystems integrated into existing systems too :)

We (Matrix) have been tracking Cobalt/Croquet on and off since the original Croquet project. Honestly, looking at http://www.opencobalt.net/, things look pretty dead currently though - no news updates since 2010? Should we be looking somewhere else?

To be clear: Matrix tries to be pragmatic above all else, and right now we're just saying "hey, use Matrix for open comms in VR/AR" (a bit like Croquet advocated XMPP back in the day). There is nothing stopping one from using Matrix to add comms into the Croquet metaverse however, and if it's alive and well I'd really hope that someone would try the experiment :)

> Honestly, looking at http://www.opencobalt.net/, things look pretty dead currently though - no news updates since 2010?

I agree it looks pretty dead, but a big bag of old, existing code still does more than a project in the idea stage :)

> Should we be looking somewhere else?

I'm not involved in OpenCobalt, I'm just someone who's also been following it for years. One of its major problems seems to be fragmentation: as far as I can tell, the descendents include:

https://en.wikipedia.org/wiki/OpenQwaq

http://www.3dicc.com

https://www.krestianstvo.org

https://en.wikipedia.org/wiki/Virtual_world_framework

Maybe there's something salvagable amongst all this?

I am not sure showing 2D videos in WebVR is actually useful or better than a standard Google Hangout. I love having a web browser open while in a video call so I can look up references, share content and other things.

Thus this feels like it is trendy (wohoo VR!), but not actually useful.

the point here is that if you view 2D video in a 3D environment with a plain old web browser on a smartphone or tablet etc, it can feel way more immersive when you hook up the accelerometers so you can look around the scene and tilt the device to look from person to person and zoom to whoever's talking etc. Obviously it makes no sense if they have goggles on, but in practice we've found it a huge win over boring 2D VC. (disclaimer, i worked on this :)

Not everybody has a high res IR camera and realtime ray tracking pipeline to share a live 3D model of the self over the network. Video will have to do for now...

Aside: To anyone finding the black-with-orange-stripes style weird/annoying, I'm guessing it is a reference to http://community-sitcom.wikia.com/wiki/Dreamatorium .

[Edit] Sorry, re: missed reference reference!

It's actually a reference to the Holodeck[1] in Star Trek (Matrix even has the same arch design as the Holodeck but which doesn't appear in the Dreamatorium). But you're not too far wrong as the Dreamatorium in Community was also a reference to Star Trek's Holodeck.

[1] https://en.wikipedia.org/wiki/Holodeck

At last, a portal site!

Oh, right.

High Fidelity is worth a try. It's like Second Life with higher resolution, except there's nobody and nothing there.

Reply on news.ycombinator.com