Show HN: Hacking on a VR desktop inside of a VR desktop [video] | Hacker News Reader

Show HN: Hacking on a VR desktop inside of a VR desktop [video]

spacepub.space

131 pointsddevault49 comments

This is awesome to see. I've been working on something similar: a remote Wayland compositor for the Oculus Quest, Magic Leap, and other AR/VR devices. Last night I hit a milestone, getting a terminal running on another machine to render inside Unity, then on the Magic Leap itself. https://twitter.com/daeken/status/1205762441285685248

I really believe that 2020 is going to be the year of VR/AR productivity, and I'm committing myself to working full-time from within the environment.

I love your enthusiasm so you got my upvote :) With that said, it's hard for me to see why some people are so bullish on VR/AR productivity. The headset seems heavy, taking it off-and-on is cumbersome, AR fidelity seems like it leaves a lot to be desired, etc.

My sister has a VR headset, and so does my cousin. I think it's a cool toy, but I'm not really sold... maybe you can convert me. After all, Christmas is just around the corner.

So, I've been working much of the time (30-50%) through the Quest, using ImmersedVR to have 3 screens from my Mac in VR. The headset is super comfortable, even for long periods of time; the only downside comfort-wise is that if I'm breathing out of my nose, it'll end up blowing upwards into my eyes, drying them out (but I'm going to be making a little buffer piece to fix that).

The fidelity is at the low end of what's usable for productivity without causing discomfort, but it's just going to get better from here. To me, $400 for the Quest is a total no-brainer; the Magic Leap .... not so much. I'd rather stick entirely to VR, but I have a kid to watch much of the time, so AR it is!

> ...but it's just going to get better from here.

Ever increasing improvements are not guaranteed. My hope is it will improve too, and ideally be done with a lower carbon footprint. But it's still an unknown.

Ever-increasing improvements aren't required; the eye has a (more or less) fixed resolution. In fact, HMDs with extremely high resolutions already exist, e.g. the Varjo VR-2 Pro and the XR-1, they just cost an arm and a leg. In a few years we'll get them at consumer-level prices instead of having to drop $6-12k.

L+2 should be the default level of metacognition. Excellent project, excellent delivery. Now I have to go put my VR headset on to go watch this video.

> L+2 should be the default level of metacognition.

What do you mean by L+2?

https://www.youtube.com/watch?v=PYylPRX6z4Q&t=221

Somewhat similar project I've been working on: Live Coding in JavaScript and VR https://github.com/m-schuetz/Fenek

Video: https://www.youtube.com/watch?v=xhGNhKVAqW0

This allows you to modify the code of the VR render engine directly at runtime while you're in VR.

You might be interested in RiftSketch:

https://github.com/brianpeiris/RiftSketch

I am! RiftSketch is great. We just rolled our own because we needed OpenGL 4.5 and access to visual studio code within the VR window. So our live coding framework is a native desktop app that uses V8 to bind javascript calls to C++ and OpenGL 4.5, and the windows desktop mirror API to render the desktop, and visual studio code along with it, in VR. Apart from the C++ core, virtually all aspects of the application can be modified at runtime by modifying the JS files and simply saving them. JS files are executed on save, and structured in a way so that they replace previously loaded functions with new ones.

Working in VR is a cool glimpse of the future. I also sometimes do it with Immersed VR (similar to VR Desktop, but works with Mac). I say glimpse, because I still kind of need to "force" myself to do it.

There are some advantages (like the big screen and more focus), but today the disadvantage will overweigh for most people (friction of putting on the headset and connecting to your computer over WiFi, the weight of the headset, lower resolution, starting to sweat over time and leaving marks on your face, etc.).

However, these are very solvable engineering problms and I'm exteremly optimistic that working in VR will be better than looking at a "small" flat screen sooner than most people realize.

This excites me. I’ve wanted to build a renderer like this for a while. The current display model is outdated. We no longer have a reason to be limited to a monitor. Either VR or AR is the visual path forward to provide us with a better work environment as we continue to become more and more of a digital workforce.

Unfortunately the current display model also has several times more resolution than any available VR headset.

That will change though - or at least resolution in both will increase to the point where it doesn't matter.

Exactly, I believe I read a Carmack article saying that you need 4K pixels per eye to get to a resolution where pixels are imperceptible (though presumably those with better than 20/20 would need higher).

That’s probably a few years away, but I’d be surprised if we don’t get there in 5-10.

Meanwhile, we need to build all the ui and OS code to drive such displays.

That said I’m not sure I’d sign up for using the current generation as my primary display device.

I wonder how much the total resolution matters. With my 27” display I only ever look at part of the screen. (Perceived) pixel density is a big thing though.

Lets not forget people used to code on displays that could only fit 80x30 character text. I'd happily give up a good % of the resolution on my monitor in exchange for an unlimited numbers of variable-sized screens organized however/wherever I want them.

I imagine one day having AR glasses with which I can "project" a vim buffer on the wall, while having huge spreadsheets that I can analyze on another wall, and basically using the physical _room_ as enormous "screen".

That kind of workspace, to me, would feel so much more natural than being constrained to sitting on a chair in front of a display.

Nice, but...

Resolution is a bottleneck. Notice only 30-ish lines are visible here during editing. How many do you normally use? And VR videos can mislead, as inside a VR HMD, only the center ~1/3 region is as clear as the video, with pixels blurring together as you glance further away.

With a custom stack and subpixel rendering one can squeeze out almost a hundred lines on some consumer VR HMDs, but it's non-trivial and not a happy thing.

Terminals in spaaaaace... remember skeuomorphic user interface design? Cell phones were new, so to ease on-boarding novice untrained users, things were made to resemble familiar real-world objects. A calendar app might perhaps show lined paper, flipped around a spiral binding, with a textured leather surround. We don't do that now - it would be silly. Because resembling the physical world is so unlikely to align with desired UI and domain affordances and tradeoffs. Windows tumbling in 3-space? For professionals and experienced users in AR/VR, how long before resembling "R"eality is design smell?

So I've used drone googles in preference to VR HMDs, for 1080p of mostly readable pixels. Shallow 3D rather than virtual world, so long hours in fixed-focus HMDs isn't eye pain. Arbitrary non-physical mapping from head and hand tracking to input, because good ergonomics is using a touch pad, not something silly like constantly lifting hand from keyboard to tap a laptop screen. Arbitrary non-physical "space" and "physics", because slavishly emulating the oppressive constraints of the physical world would be... just why? Intensively avoid immersion. And so on. Arguably not vR. But is "R" really the best possible professional use of AR/VR hardware?

Nreal light 1080p AR glasses recently became available for $1.2k dev kit preorder. Which is months late, but recent PR is still quoting $500 "early 2020" consumer availability. We'll see if pixel blur and alignment is good enough to render a small font from corner to corner. If so... I hope to be using my laptop screen rather less soon. But not to see windows floating in spaaaaaace.

EDIT: Oops, I just noticed this was a ShowHN, rather than a generic post. So on a more upbeat note... it's great to see the envelope being pushed, with non-toy real extended demos, inspiring people, and encouraging them to reflect on the very non-trivial changes which are inbound. And PeerTube worked pretty well.

This will be very useful when traveling as a digital nomad (again). Instead of being limited to a small laptop screen, just use as many screens as needed, with no extra luggage.

I would like to see a review of what it’s like to work on these environments. My sense is that the resolution and weight are real problems, but maybe that isn’t true anymore.

There is something appealing about being able to boot up a multi screen completely isolated environment for deep work.

One thing that I'm not sure is generally understood is that Samsung etc. are advancing folding-screen phones, and MS is jumping back into phones with a dual-screen model, for just the same reason that Apple is apparently working hard on AR—and the reason isn't simply to have something new to sell smartphone users. It's because the usability of the smartphone as an all-round personal computing device is now massively bottlenecked by its small screen, and if you simply make that screen bigger then as soon as you go beyond phablet size you simply end up with a tablet, too bulky to carry all the time in your trousers pocket. And when the alternative is an overly-small screen too close to the eyes then (present and near-future) AR/VR drawbacks like low pixel density are not so bad in comparison. (One snag here is the bulk of the AR HMD itself: we'll have to see what Apple has done on that front.) Laptops have that problem to a lesser extent, being unequivocally on one side of the screen-too-small/device-not-ultraportable tradeoff, but AR/VR is going to displace (or at least supplement) the laptop clamshell screen too, and it will eventually come for desktop monitors as well.

Since others are sharing their VR productivity apps, I will do the same:

I wrote the first WebVR framework in the world. With it, I built a very simple live-programming environment: https://primrosevr.com/demos/editorVR.html

That was 2, almost 3 years ago now. I'm no longer working in WebXR, but I am still working in VR productivity. I'm currently working on an app for language and cultural learning in VR.

Unfortunately, there is nothing public to see, yet. Though we did just give our first semi-public demo last week: https://dlsdc.com/blog/dls-hosts-tenth-annual-open-house/

It’s a good experiment but from a productivity point of view, I don’t get how it’s better than a regular desktop (especially considering you have to wear those big goggles!).

But maybe that wasn’t the point.

Anyway, nice to see it in action.

I'd borderline use this before I tried to work in an open floorplan office.

That said, it's too early on the hardware for resolution and size/weight.

If you want an intuition about why this will be compelling in the further future -

What is a memory palace and why is it so effective?

It appears that we encode a great deal of our thinking, memory, and conceptual relations in a spatial way in the brain.

Maybe a 2d laptop screen that portals into a world of constantly rearranging 2d windows doesn't give us a strong or intuitive sense of place?

What if your computing environment supported or even enhanced this type of natural way that your brain worked?

https://en.wikipedia.org/wiki/Spatial_memory#Virtual_reality

https://www.eurekalert.org/pub_releases/2019-11/cuso-snt1107...

https://en.wikipedia.org/wiki/Place_cell

Imagine if program windows weren't constrained to the flat planes of your monitor, but could be anywhere in your field of view at your desk. That's the dream, and this is a stepping stone I think. Soon enough the VR headset will be slim and ergonomic, seamlessly combining reality and the virtual workspace so you don't feel separated (if that's what you want).

Potentially, you get to be more portable. Big screens are better (to some limit), but you can't carry 24-27" displays with you. With this, all you need to carry are the goggles, keyboard and mouse.

I feel dizzy just looking at the window changing and also, I already don't look at my laptop screen preferring to switch desktops on my external monitor, so the idea of moving my head to look at stuff doesn't appeal to me. I guess VR as a screen replacement isn't my cup of tea (at least not yet, maybe it'll change in the future).

That's without even factoring in the fatigue from wearing the headset (both in terms of weight and having something strapped to me, eg I can only wear headphones for so long, even without sound, before my ears feel tired). I also don't like the idea of shutting the world out, especially since I like to use paper/white boards.

Agreed. I want this to be an improvement but real productivity has little to do with how many screens you can view simultaneously.

Attention is a finite resource.

In my experience, having some apps physically out of view (like OP's IRC and music videos) actually help with focus sometimes compared to a traditional monitor setup, where you always have a visual reminder in your taskbar or when you alt+tab to check in on IRC, Slack, HN, etc.

I guess you could mimic that experience by e.g. hiding your taskbar, but "out of sight, out of mind" definitely applies here w.r.t. focus, and it seems like a 360-degree space offers unique ways to keep things fully out of sight while still readily available.

Someone should correct me if I'm wrong, but it looks like he's wearing a first-gen Vive, which is surprisingly light (especially when you swap the tether of a cord with the wireless adapter).

I've got nearly 2500 hours in VR and sometimes spend up to 8 hours a day with it on, and it is way less big/bulky than it looks.

I put giant googly eyes on mine, even. :)

I don’t think this would work very well on the Oculus Rift S. I’ve tried using BigScreen to navigate webpages and it’s just not that easy to read normal sized text. Maybe it’s better on the Valve Index?

It is, but it's still not quite there yet for productivity, I feel. We'll see what the next generation brings, but GPUs better keep up, as even the resolution of the Valve Index is already really punishing on the hardware.

I've been working on the Oculus Quest (which has the same resolution as the Index) and it's the lowest resolution I find functional. In my case, it's just barely good enough to use for long periods of time, without absolutely killing my eyes. Pretty excited to see what the next generation looks like.

I've found bigscreen pretty terrible for this, try something like virtual desktop.

If bigscreen has the option, curved displays are also a great way to preserve clarity, as the lenses start to distort near the edges.

Cool! I love seeing tools like this. I've also done a bit of research into VR desktop experiences https://twitter.com/trevorjbaron/status/1197043776197251074 mainly built using webGL + other web tech and running on Quest.

This brings us one step closer to Heavy Rain's AR desktop: https://youtu.be/SsQT3mbvVWY

I wonder if it's possible to get the same outcome of the game using such thin glasses, though. The current VR headsets look a little ridiculous to me :)

How cool! I remember Mozilla announced some work on improving text rendering under 3D transformations, do you think that could be integrated? https://github.com/servo/pathfinder

Awesome!

There’s a quiet little subreddit devotee to this idea https://www.reddit.com/r/HMDprogramming/ They’d really appreciate this.

I went ahead and posted it there... hope you don't mind Drew

https://old.reddit.com/r/HMDprogramming/comments/eandil/hack...

The text looks really good in the terminal in the video. Is the display as good as the video capture?

Also, what is the hardware used for the video card and HMD?

It's because it is huge. Compared to a normal desktop monitor, he is looking at like 30 point text. If you were to render text as the same size as a normal monitor, it would be illegible on a vive (which is what it looks like he is using). The thing is, when you have near infinite surface area, having big text is no big deal.

I was wondering more about screen door and texture aliasing effects because of barrel projections, etc.

Thx for the note on the vive

It'd be interesting to see how the eye damage/strain caused by reading small text on a big monitor compares to the eye damage/strain caused by reading big text on monitors inches away from your eyes.

Can someone explain the appeal of japanese pop music? I mean this in an entirely non-sarcastic way, and I'm looking for a serious answer.

It can't be the lyrics unless you know japanese... maybe it's just me but I don't find the melody appealing at all (even from a more general POV where I acknowledge the wide range of melodies from country to classical to death metal).

Is there some part more to it that goes beyond melody or lyrics? I'd say something similar about extremely hardcore death metal which is essentially just noise, but I imagine to someone who has that personality type it's almost cathartic, so even that I can understand the appeal, but not japanese pop music, especially while coding lol

Not understanding the lyrics can be part of the appeal, because it makes them less distracting.

Understanding a little can make embarrassingly cheesy and saccharine pop lyrics take on a profound poetic quality.

Growing up on anime and Japanese video games is probably a big contributor to finding the melodies appealing.

Also, dude's got a giant concert video in view pretty much constantly, so image is probably part of it.

hope this helps

Ok thanks, the japanese video games + anime answer puts it into perspective, I hadn't thought of that. I could see how growing up with either of those would mean listening to that music then would bring good memories or feelings back.

It's not nostalgia, I didn't grow up with Japanese culture. And it's not that I can't understand it - I can. I just like the music. Japan has the second largest music industry in the world after the US. They produce a huge, diverse group of interesting music which stands fine on its own merits.

Reply on news.ycombinator.com