Moving the Linux desktop to another reality
collabora.com
collabora.com
Also Valve tbh have dropped a lot of their more interesting tech. I had one of their steam links which isn't even sold anymore. The steam link was actually pretty useful for steaming anything on your desktop, I am sure you can do the same with a Raspberry Pi and some scripts but it worked pretty well if you hooked up a mouse and keyboard.
I've found that the steam link apps for other platforms (ios/android) work very well. I'm not sure Valve needs dedicated hardware for this anymore.
I have a steamlink in the closet, but being able to do it in software is nice.
I never got a hold of the Steam Controller either. It never seems to do what it's supposed to. I'd love to be able to switch between Controller mode and K/M mode from the controller itself, but it never works. I also hate that it can't seem to just map itself to act like an XBox controller. Generally what happens is the game has no idea what to do with it, and I'm stuck until I get up and go to the computer itself.
I probably don't know enough about it, but I've had it for years and I just can't be bothered.
This is more common than you would suspect
Actually you don't even need to write your own scripts. There is an official steamlink package available for Raspbian. I just set up a Pi3 to run as a dedicated steamlink the other day, and it works fairly well.
Just run:
apt install steamlink
steamlink
@reboot steamlinkAnyone interested can get the Debian package here:
http://archive.raspberrypi.org/debian/pool/main/s/steamlink/
The resolution kills that notion for now, but someday it might become a cost effective setup.
Add a pair of noise canceling headphones and you can pretend your tiny desk is a private office! It even has imaginary walls to hang things on!
- Full clarity along the lens
- Double the resolution of what the current VR has
- Foveated rendering (to be able to handle the high resolution)
- Dynamic focus
- Eye tracking (to get dynamic foveated rendering and focus)
- More comfort (lighter headset, improved finger tracking, better weight distribution, better foam, cooling etc.)
But if you want to bring it into a fully virtual world instead of on top of the cameras, then you can't see your keyboard, mouse, or fingers. I can blindly touch type well enough to get by, but for it to see any adoption you'll need to get the input devices visible in VR.
xrdesktop looks very impressive. We've been working on our compositor (in particular: XWayland support) for quite a while. Getting a prototype up and running was relatively easy. But it has been incredibly difficult to get everything actually stable/usable.
Congrats to your team.
I really like multi-monitor setups, and in the limit this can far surpass physical hardware and space limitations. You can scroll more than 360 degrees for a near-infinite workspace. (Just spin around a hyper-sphere repeatedly.)
I really want to see this tech succeed. I'd look like an idiot using this at work, but I'm already sold.
I'm forced to use a Mac at work -- do you know of similar tech for MacOS? I'm gladly going to investigate this at home.
I don't know of any similar tech for MacOS at this time.
I use four monitors at work and two 4K monitors at home. A high-resolution VR setup gives you infinite workspace. Couple that with intelligent auto window management and you can forget moving windows around EVER again -- this is the ultimate mouseless experience that a tiling window manager can't even get close to.
This is a huge productivity gain. The micro context switches that stem from moving between windows and tabs (or worse, virtual desktops each with windows and tabs) goes away. Workspaces become spatial, and you can leverage your brain's innate spatial reasoning to find and organize things.
Physical project workspaces are laid out logically. We've been constrained to screens and haven't had the freedom of doing this for our virtual ones.
Plus VR strains your eye really quick and that comfortable to wear for long periods of time
A real keyboard that you can feel and see as you type on. Both hands would need to be tracked (in addition to the keyboard location) using something like leap motion not by wearing some gloves/controllers or attaching anything - no one is going to do any amount of typing on a virtual keyboard.
When there's a good solution to this I suspect quite a few developers will give it a go when they want a fully immersive in-the-zone experience.
I think the Index's cameras would be sufficient. Alternately, you could place tracking pucks (or just controllers) on either side.
Is it? Are the lenses that much better? My Lenovo Explorer WMR is 1440^2 px/eye, and thus has a similar angular resolution to the wider FoV Index. Both are RGB, not Pentile. But I find each eye gets a central circular region something vaguely like 300 px diameter where subpixel rendering is worthwhile (I've a custom render stack), and less than 600 px before the lens blurs pixels together. That's 1980's VGA-like resolution. And it gets even worse as objects move in from infinity and thus out of these regions. If you disable the "oh, its an HMD, not a normal display" feature the OP speaks of, you can easily throw up a pixel-aligned test pattern image. The results are... blurry.
So I use DJI's 1080p drone goggles. Readable 5 pt fonts, almost corner to corner. With shutter glasses, because DJI didn't expose the two panel feeds. :/ And reading glasses, because DJI's lenses... sigh. It's a placeholder kludge.
Nreal's 1080p AR glasses could be available in a couple of months.
> the critical show-stopper is not being able to use your hands to type onto a real keyboard
I've been using my unremarkable laptop keyboard since Vive, by doing pass-through AR with a camera gaff-taped to the HMD. With a browser-as-compositor stack, it's two-video-dom-elements trivial. Though I've heard with a SteamVR stack, pass-through latency can be a challenge.
Hand tracking is more problematic. It's a poster child for acquisitions wiping out market infrastructure. The sole hand-tracking survivor was Leap Motion, which fumbled its couple of Apple acquisition attempts. It's been Windows-only, stalled, kind of crufty, and now has finally been acquired, with uncertain consequences for future availability. And one can kludge. But it isn't pretty.
> When there's a good solution to this I suspect quite a few developers will give it a go when they want a fully immersive in-the-zone experience.
And shallow 3D seems to have interesting possibilities for IDEs.
Yes, the lenses on the Index really are that much better. The difference is enormous.
I'm still not sure the resolution is quite high enough that I'd want to use a VR desktop environment, but I absolutely could now, reasonably comfortably.
With something like that integrated into rendering libraries, I'd love to have the field-of-view of a motion-sensitive environment. (I don't actually want pseudo-3D, just a high-res 2D display attached to orientation sensors.)
I'm eager to see where this goes, and may need to start shopping for a head-mounted display.
And then there is the issue with holding 3d pointing devices for prolonged amounts of time. This is taxing upper body strength if you do. The weight of thenheadset itself is also straining the neck muscles.
All of this taken together usually means that I want out of VR after about 2 hours. I barely have sessions that are longer.
I don't think I'd like my desktop to have a lot of 3D features or require a 3D pointing device. What I'm imagining is essentially a giant 2D workspace where I can still use a Trackpoint.
The large screen concept is a bit problematic. A large planar screen distorts the portions you're not in front of because you're only seeing them at pretty extreme angles. A cylindrical screen distorts projected content, e.g. photos or 3d graphics displayed on it. A better solution would be to treat each window as a flat rectangle that can be freely positioned in 3d. So you can arrange the most important window around you in ways that you can conveniently look at them by turning your head and you'd still look at them dead on. For implementing that, a 3D pointer is by far the best option.
I use DJI's 1080p drone goggles. With DIY shutter glasses, because DJI didn't expose independent panel inputs. :/ It looks like a 3D TV screen, not VR, but you get readable small fonts from corner-to-corner (almost, depending on whether you also need to use reading glasses with it). But with batteries, they indeed aren't light weight.
Nreal's AR glasses could be available in a couple of months.
> problematic [...] angles [...] positioned in 3d [...] arrange [...] around you
The skeuomorphic design phase with phones was a short-lived part of onboarding an untrained user population. For non-novice professional user interfaces in XR, I suggest it's... silly. :) But oddly reoccurring, perhaps because of game tooling's availability?
When you use office apps, there's little attempt to maintain physical-world properties like say object permanence. Delete a word, and it "goes someplace, because it can't just vanish"? That would be silly. There's no need to emulate physical paper, or pens, or rubber erasers, or cut-and-paste with an xacto knife, paper fragments, and gooey paste from a jar. It can sometimes be useful or artistically fun to do so, but that's a UX choice.
Or say you're looking at a whole-desktop "window", placed uncomfortably close and large, to be readable despite current HMD's limited angular resolution. When you turn your head, how much does it move? With what velocity profile? And with what alterations in appearance? Skeuomorphism suggests it appear as a real physical object, nailed in space. I found I preferred a quantized 2-3x displacement, to reduce head motion, and enabled it by transiently increased transparency (including "comfort mode" clipping from fov), with slightly delayed motion and added jiggle, so other visual cues could dominate wrt balance. And the shape distorted to maintain pixel alignment, because given my weak gpu, alignment was worth more than temporal antialiasing when reading small fonts. YMMV, as just what combinations of cues are fine vs intolerable varies so much among people.
The common thread is aphysicallity. Skeuomorphism can be useful when onboarding novices, but is later often largely abandoned. As with phones. With the current focus on immersive games and novices, that second part tends to be neglected. I joke that while I use XR hardware, I'm not doing XR, because being focused on non-game non-novice UX, and using a custom non-gaming stack, I've no allegiance to the "R", to Reality. Resemblance to reality as UX design smell.
You are talking about an simulated non-physical world that makes no semse at best and I'm certain that it is just a nausea and disorientation simulator for almost every single user if I understand your thoughts correctly. The thing with VR is that it messes with expectations about the physical world that people process subconsciously. Break them ever so so slightly and people react involuntarily to it. There is no consciously controllable component to getting uncomfortable in VR.
One curious personal observation: out of the ~10 Asian coworkers who tried the Vive, all experienced some degree of motion sickness related discomfort.
You could use a real mouse and keyboard, if you're presented with some kind of virtual workstation. And even for navigation outside the workstation, of your broader "environment", 3D video games have been using a mouse & keyboard to navigate complex interfaces & environments for a long time, with more than a little success. Granted that means you still need a surface of some sort, but I think non-touch-typing text input inside a VR space would be kinda hellish anyway, so you're at least gonna want a real keyboard regardless.
Positioning objects in 3D precisely and conveniently without a 3D pointing device is hard, too. The best option for that is a mouse and crutches like handles oriented along coordinate axes. But that only works because the mouse pointer has a reference plane it moves on (the projection plane for the 3D view). In VR, this plane is tracking the head precisely. You cannot reasonably put the mouse cursor on that plane. You would need a different reference plane and then you're back to all the problems that 3D input on a 2D screen has. And a positioning objects in 3D with a keyboard is even more tedious.
They wrap around your hand, so you can type (ish) and not take them off.
I'd think it would be fantastic to physically walk around to different monitor/window/server setups (in VR), though, and that'd require dragging around a rolling standup desk, or figuring out how to type based on finger positions from the controllers (which I'd imagine would be decidedly not great).
Another question to ask is, given hardware with 3D tracking and display, how might you usefully tweak your existing tools to leverage that?
Rather than reinventing wheels, languages, keyboards, etc, I like the approach of augmenting what exists. And I use emacs a lot.
TensorFlow posenet-based head tracking is easy.[1] And wiggle 3D, and anaglyph shaders, aren't hard. And making DIY shutter glasses (arduino, phototransistor, and Adafruit light valves) was surprisingly simple. So my generic thinkpad screen can do jittery, washed-out, or flickery 3D. And electron/node.js/v8 can serve up low-latency video of the desktop, to be sliced and diced. So an app can provide two or three eye views, in separate windows or time multiplexed, and it can be shown as 3D. When last I tried to do that separation in emacs, I timed out, still unsure if there was a viable path. Thus my query.
Another thing one might try is to overlay graphics. An emacs could have an arbitrary backchannel chat with the compositor, and draw 3D lines and such on itself. And of course control its own window positions and orientations, fragmentation and coalesce. ;)
If you have red-cyan anaglyph 3D glasses, you might enjoy toying with https://atom.io/themes/anaglyph-syntax - but trading color space for depth is a steep price.
> What would you use this for?
So basically for exploration. What might our tooling now look like, if we'd had 3D display and tracking in widespread use for the last two decades?
[1] https://storage.googleapis.com/tfjs-models/demos/posenet/cam...
AR HMDs trade narrower FoV for angularly smaller pixels, and you can use "all" of them. But the pixels may be poorly aligned between eyes, as with Project North Star. Maybe Hololens 2 (with different a-v issues). I can't comment on Magic Leap. I look forward to trying next month's/year's Nreal glasses.
Media-viewing and drone HMDs offer a screen at some distance. Some are stereo 3D. Some are 1080p. Few have head tracking. You can add it with Intel t265, Zed, Structure Core (no linux), or DIY. Many have a "diopter" adjustment. They've varied build quality, and supported nose sizes.
Some months back, I failed to find a 3D 1080p HMD, and so currently use mono 1080p DJI drone goggles (discontinued cheaper white model), with anaglyph (crushing the color space to reduce eye strain) or DIY shutter (flickery 30 or 20 Hz) to recover 3D. :/ Usually with reading glasses, despite my old eyes being nearsighted, which costs some edge pixels. DIY optical tracking, so no temporal antialiasing. I'd not recommend it.
My current hope is for a Q4, 1080p, 50-ish degree FoV, 3D "screen" HMD (media-viewer or AR). With either head tracking built in, or COTS. It may need a Windows box in support, especially for hand tracking. I'll likely stay with a simple custom browser-as-compositor stack, for low-cost pixel-precise control, but a mainstream VR or AR stack might work too.
The only other drawback of 16:9 is that in order to get an acceptable height, it makes the laptop too wide. My solution is to use clip-on reading glasses over my regular distance prescription, when using a high res laptop screen.
Oh god... where's your anisotropic filtering? That looks like a screenshot from Doom.
Chromebooks are great for what they are. Fantastic for people that just need access to the web. They're pretty much immune to the technically-challenged.