Live Coding in VR with Oculus Rift, Firefox WebVR, JavaScript, Three.js [video]
youtube.com
youtube.com
This fills me with almost childish glee. This is a virtual reality, where you can pop up a "god window" and create/mess with a world around you. Even just what's in the video, making some cubes move and change colour is wonderful. It's fun!
Remember the early bits of amazement at programming? Where you could make the computer say your name or draw a square? How much cooler would that be in VR?
Croquet was this (but without the Oculus) in 2007. I miss it, but it went nowhere, but.. that's Smalltalk for you - seems to scare people for some reason.
i argue against people calling the oculus virtual reality (i)
that child like glee is warranted in a vr scenario, but what you are seeing demo'd here is digital stereoscopy
.
My problem is that it lacks imagination. Let's just take our text editor over to the VR world. Not come up with a new method of interacting with code utilizing VR.
Yes, and I think live coding is pretty cool.
> but I don't see the revolution here yet.
Neither do I, but I don't think it needs to be. It certainly didn't seem to be sold as that. That was part of the reason for my post actually, that people are alternately seeing this as the next big thing or falling short of some incredible goal. It's neither, it's just cool.
> My problem is that it lacks imagination.
Well feel free to create something else incredible, and I'll be happy to find that awesome as well. This doesn't hold anyone back, it doesn't stop anything and it isn't being billed as TheNewWave(tm), it's just an interesting experiment combining a few different bits of tech together.
It's a drastic shame to see people poke holes in something just because it isn't world changing.
Also I am working on something on my own. Which is why I thought I'd give feedback. I encountered the same issues myself.
I'm not sure if that was aimed at me, but if so you can't downvote replies to your own comments (at least I can't, maybe there's a higher karma threshold where you can but I doubt it).
I agree with your overall sentiment, but I also have an observation: I have the DK2 -- and while I haven't had a chance to play with this yet, I have been playing a little bit of the Elite: Dangerous beta.
The revelation there, is that working headtracking makes working with a huge "display" feel very natural. You can only look at one (or a few) things at once anyway -- but with head-tracking you don't have to worry about lining up five 30" screens in order to keep track of everything -- having it all (sanely) mapped out in 3d actually works just as mundanely and boring and well as you'd expect.
And that in itself is kind of a revelation. Suddenly you have almost unlimited screen real estate, that is even more naturally and intuitively accessible than multiple desktops (and I'm generally a great fan of xmonad-style tiling, no-full window managers with easily accessible multiple workspaces).
Another "wow" factor for Elite was the fact that what seems like a kind of silly gimmick on a single screen -- that "holograms" pop up with info when you look at them -- actually works in VR.
The one thing they did botch in Elite in the previous beta, was font-size -- the DK2 leaves "half-hd" per eye -- and that's not really enough to read "fine print". They sort-of fixed it in the new Beta just out -- but now the UI for space docks is "too wide" -- the text is more readable, but the "holo" should've been placed "further away" from you, so you don't have to move your head from side to side to read a line of text. I've yet to try it plain 2d -- I suspect it works fine on a regular screen.
Anyway, the point is that even a completely boring an unimaginative use of VR for programming can actually be a minor revolution. Fully working gloves/hand tracking would be better -- maybe with just a "dummy" keyboard for touch input (I'm not convinced "drum on a tabletop, visual-tracking keyboard" is actually that usable for touch-typing -- at least not when your back-tick is at shift-two-places-right-of-zero and opening-curly-brace is at meta-7 etc -- as it is on a Norwegian layout).
Finally -- immediate feedback certainly is nice -- and this is where the "big screen" effect of VR can really help. You can be effortlessly aware of both your evolving 3d model, and the text-representation of your code -- without having to have two screens, or switch between virtual workspaces.
You make some good points and I'll have to investigate further once I get the headset. Currently using the SDK and it just isn't the same.
I kinda like the "grab object & kiss to save" gesture: it wouldn't make sense in 2D but feels natural in VR, although "grab & push object in your brain" might be easier implement to represent a save function.
Someday there will be something like Second Life with Oculus support and a decent programming environment, and that will be awesome.
Note that in the video there's just one editing window, in an IDE obviously there could be editing windows and references all around you. You would never need a mouse as your head movements convey positional information, though you can use a mouse of course, and with good keyboard controls I think you can have an editing space ten times as big as the traditional dual 27" monitors give you.
No more hiding buffers in tabs, no more disruptions of flow as you're scrolling through your buffers looking for the right one. You just access the visual memory in your brain to intuively remember where you left the buffers you use.
Also, the idea that this would somehow be bad for your neck sounds ludicrous to me, how is staring for hours at a stationary rectangle with miniature information more natural and easy for your neck muscles than having information spread around you in even spacing at a comfortable virtual distance? I think it's going to be a relief for nearly everyone.
To a degree. But I when I work I have two WQHD monitors (2560x1440) with one turned on its side so I have one text editor with a vertical resolution of 2560. When ever I work on a laptop, even an HD laptop, I miss this setup so much as it feels like I am drinking through a straw.
Because of this I am unsure if VR, which is going to be much less than what I am working with now, is going to be a serious replacement.
The display technology and software is still a long way away however.
OP here. Well, I'm a bit more optimistic. Oculus now has ties with Samsung's display manufacturing, relationships with Nvidia and Facebook's cash. I wouldn't be surprised if they become the driving force behind high-density 5" displays and graphics horsepower in the next few years.
I think pixel density is what people want - to make full use of that 3240p you would have to move your head quite considerably, to make use of a large monitor from 75cm away doesn't need nearly so much head movement.
It's what I want and am hoping for, but the current DK2 is in no way usable for this
edit: ah I see, I hadn't factored in the warping of the optics, if you're correct then the effective resolution would only be around 900p so I'm being too optimistic :)
There is a virtual space 10 times bigger than 27" monitor, but it's only in my head, and I have no need to see it in 3D space.
But that's because I work in this space every day, I know all filenames by memory, I always know exactly where I wanna go, and type cmd+t followed by filename automatically.
This VR idea is like flying through places, but you don't need that if you can teleport.
I can't imagine this being useful in day-to-day programming, but definitely would use it for discovering an unknown project.
It's not, didn't you watch the video? You don't fly, you sit still. Even if you would want to move your virtual presence, I'll guarantee you that the computer can teleport you faster than your brain can, not that I think that would be comfortable..
Anyway, I'm glad that you're happy, so by all means stay happy.
And then there is the problem of you brain can think about things that aren't actually done, yet. Computers can certainly help in this regard, but they are more about sharing something that was done in a different way. Consider walking through a house. Sure, a computer can let you walk through one that hasn't been built physically, yet. I'd wager skilled architects can imagine walking through one that hasn't even been built virtually yet.
And, finally, there is the problem of thinking about data structures that are literally gigabytes in size. I have a hard enough time thinking about ones that are megabytes in size. I'm not entirely sure accurately visualizing large instances would really even help.
once you're familiar with your dev environment, you don't need to have it floating around in VR space. Its all memorized pretty much, instant access, L1 cache. so to look at something in VR you turn your head and the view pans. To look at it in your brain, its just there, instant lookup.
one requires a transition in space to access information (flying) the other does not (teleportation)
Low-pixel displays can't replace high-pixel displays (yet). It's all about the speed your head can move, vs. the speed your eyes can move, vs. the speed your display can refresh.
As a simple test, try holding your head still while moving your eyes back and forth ten times, from one extreme of your FOV to the other. Then try moving your head back and forth ten times to see things beyond your FOV on each side. Two observations when I try this: (1) the eye movement was much faster than the head movement; (2) the head movement was much more tiring.
In fact, eye movement is the fastest thing we can do, and even uses special circuits to reduce latency:
"Saccades are the fastest movements produced by the human body. The peak angular speed of the eye during a saccade reaches up to 900°/s in humans; in some monkeys, peak speed can reach 1000°/s. ... Under certain laboratory circumstances, the latency of, or reaction time to, saccade production can be cut nearly in half (express saccades). These saccades are generated by a neuronal mechanism that bypasses time-consuming circuits and activates the eye muscles more directly."[1]
Head movement apparently maxes out at 500°/s or so, and that's not comfortable to keep up for long. So it's at least twice as fast to access high-resolution information by eye movement as to access the same information at lower res by turning your head. That's the tradeoff between using a fixed monitor at your max comfortable resolution, and using a lower-res head-mounted display.
At the same time, a fixed display (for now) allows you to move your head more rather than less while reading comfortably. A head-mounted display has to update the display in response to your head movement, and a fixed display doesn't.
As a test here, see how fast you can move your head back and forth while reading text on your fixed display. Then see how fast you can move back and forth while reading on your DK2. The second will be slower, because the Oculus has to impersonate a pixel on your virtual display by juggling which physical pixels are used to render it as your head turns. In addition to further throwing away resolution by aliasing your virtual display onto your real display (the virtual text has to look sharp regardless of how it happens to overlap with physical pixels), this means the refresh rate is a limiting factor on how fast you can turn your head and still read. The text which was crisp while your head was still is physically blurred before it even gets to your eye, because the Oculus can't quite keep up with where it should be. That stops being a factor at, say, 1000hz, but we're not there yet.[2]
So on a fixed display, you can actually move and stretch your neck and turn your head without affecting the reading experience; on a virtual display, those movements will blur the text you're trying to read.
I'm very excited about all of the things VR can be -- just saying that one thing it can't be, yet, is a replacement for a high-resolution monitor when you're trying to quickly access lots of text.
[1] http://en.wikipedia.org/wiki/Saccade [2] http://blogs.valvesoftware.com/abrash/down-the-vr-rabbit-hol...
need some new/better input devices in addition to jey oard to manipulate the objects in the space.
does the oculus have a "axis lock" feature of some sort? foe instance, lock X so that any head movements only change the users camera along x?
Though as the other guy brought up, I guess it may not be the most natural way to read text if you have to actually move your neck, but it may be decent enough that you could adapt to it. And thus, you could write code on the train/bus to work from your phone (at least, until someone mugs you and steals your expensive phone VR setup heh).
Text had to be very large, so only three or four lines fit on the screen at a time. After hearing people praise the DK2, I thought it would be basically seamless, but it was jarringly low-res.
The experience was great, very natural, but the pixel density left a lot to be desired.
FOV is normally diagonal, so for 800x600 we have 1000 "diagonal" pixels, and for the rift's 960x1080 (per eye) we have 1444 "diagonal" pixels.
So that works out to:
1000/40 = 25.0 pixels per degree for the z800
1444/100 = 14.4 pixels per degree for the rift
In fact, it's even less for the rift because the DK2 doesn't have a standard RGB display but pentile pixels, which means you only have the full 1920x1080 for the green subpixels, the red and blue are reduced since humans are less sensitive to those colours
However, in addition to the low resolution in the current Oculus dev kits, text rendering is definitely hard to do in VR. How do you do crisp sub-pixel rendering when your face is ever so slightly tilted? Something that will have to be solved in future software or by sheer pixel density.
That said, it's early days for VR and the future is bright!
But... the video shows a very specific example of programming where you are basically designing a map or a level. As soon as you are programming something else than a simple 3d game, it becomes less useful.
The major problem we are trying to solve here is managing windows, but I argue that normal screens already do that and you don't need VR.
A large screen or two plus using alt-tab and virtual desktops can completely and better replace virtual screens in VR. You have to move in the VR world to access information. On the other hand using a normal desktop where the information is represented abstractly( where in VR it is a physical object ) can be accessed instantly with a simple and fast key-combination. I simply don't see which programming activity could you do better in VR than in a normal desktop.
Using this app is comfortable but VR will definitely need higher resolutions for anything more serious. It's not too far in the future though. If things go well, I expect VR will be the driving force behind ultra-high-density 5" screens in the next few years. Or some entirely new display tech (retinal projectors?) will replace it.
I'm trying to think of a good example where VR could be used but just can't.
It will probably be fantastic for stock brokers where their information relies on quick glances over a lot of realtime graphs/tables and if you look at their monitor setup today( +4 monitors ) the extra space will work for them. But when you code you are heavily centered on a single window and the rest doesn't really matter.
For times like this you might want a completely black/white background, a forest clearing with very gentle soundscape or a quietly psychedelic lightshow/music all centered around your code window.
Some people's VR world will be serene and spare and some people's will be chaotic and full of the energy of a city street. It will be up to the user.
Some people will use the fact that everyone's brain benefits from both of those extremes at different times and use that to improve their cognition.
A company would never buy an VR rig for every employee just to maybe boost their productivity by an unknown margin. Also the learning curve would set them back by an unknown margin and some might have problems adapting.
Using your example, VR is useful, but only as an edge case where VR measurably help a person.
Think of it as switching every keyboard in the company for a Dworak version. That will not meaningfully affect your average programmer's productivity, as their typing speed gained from a faster layout gives only diminishing results. Dworak only makes sense for a programmer that really benefits from >100 words per minute, and most really don't. The bottleneck it the programmer, not the input. The same goes for VR, unless you have a special case as I have mentioned before.
Let's say you type 300 lines of 80 characters of bug-free(!) code per working day of 8 hours. That is an pessimistic estimate( for my theory ), average is much lower.
Let's assume the average typing speed is 60 words per minute. That gives you 83% of the time not spent typing(!). Doubling the speed to a very fast of 120 wpm, only gives you 92%.
Just guestimating, I would need 11 or 12 27" screens to fully utilize the area I can see without moving my head. With gentle neck motion included I think that number triples or quadruples.
If you switch windows every 60 seconds, this gives us an improvement of ~8%.
Table:
300s -> 1.5%
120s -> 4%
60s -> 8%
30s -> 16%
15s -> 32%
I have no idea what my average window switch time is. I should measure it sometime. I doubt it is under 30 seconds where the VR actually makes a meaningful improvement.
If you take an average of 2 seconds for non-VR and a skilled user with keyboard shortcuts and virtual desktops, VR advantage becomes insignificant.
Table:
300s -> 0.6%
120s -> 1.5%
60s -> 3%
30s -> 6%
15s -> 12%
If you compare a specialized window managing software versus the VR equivalent then I don't see any benefit towards the latter. The latency is the same. The only difference is the abstract windows management versus spacial of VR.
I think programmers can handle the non-VR solution just fine. I really hope it becomes better though; the resolution is currently just too small.
Any suggestions for actual programming? You know, like; classes, boring code, loops, i/o, algorithms?
You'll soon build a counter strike level by walking inside it and sculpting the world around you. How cool !
Everything from design, analyses, testing moves into the VR environment. Already CAD/CFD/CAE computational models are saving huge costs to companies. Throw in VR and we are that much closer to a tangible, collaborative environment. Very very excited!
this demo is cool but i'd argue impractical
be careful getting excited over demos of technology that add a layer of abstraction from the tech direct
ever seen a photo of a painting? ever read a book you loved and your friends hated?
i'd suggest you try the oculus as you would use it: in your home on your box; before spending any money on the device
also, you are blind to the keyboard with the headset on
i feel oculus caries a mythos around it that is very difficult to obviate if you have yet to have worn one.. i'll link to an old comment i posted to carmacks keynote at the first ocucon : https://news.ycombinator.com/item?id=8347450
essentially i have stopped referring to oculus as vr, oculus is digital stereoscopy, and being stereoscopy the oculus is exploiting an optical illusion and as such, i believe, a flawed direction for the touted outcome: virtual reality
this demo: http://www.phoronix.com/scan.php?page=news_item&px=MTcyMTI ; is what encouraged me to shell out for the dk2
the experience failed to deliver on my expectations, though at the time i thought my expectations were practical and rooted in evidence, of course now i realise those expectations were irrational and that evidence was secondary being derived from this video demo
Of course stereoscopy is part of what the oculus does (although there's also head tracking, so talking only about the stereoscopy is misleading). You say it's a flawed direction, so what's the right direction then?
i suppose it depends on your definition of virtual reality..
but i think the verbiage of vr is very misleading
my position is that if oculus was called: oculus digital stereoscopy; the tech would lose a lot of its hype
as for the tracking it is certainly the best feature of the headset, again with the goal of calling the thing what it does i refer to the 'head tracking' as 'inferred spherical screen encapsulation'
what the head tracking is doing is windowing your vision to the size of a fisheyed cell phone screen and then inferring your position in a 3d space and displaying the contents of that field of view on the screen, refreshing the content of the image in regard to the movement of your neck, giving you the impression that you are in a sphere of flat displays looking through fisheye glasses with blinders with a forced forward stare
when i put it on the restricted periphery and the inability for me to use my eyes to look around was the most dramatic realisation
but that is just dismantling the vr hype, actual stereoscopy is flawed in its own ways..
peoples eyes differ in distance apart, the oculus is solid plastic, certainly the error is measured in mm to fractions of mm but that is still error that your brain now is required to correct,
also the goggles press against your face tightly to seal out distracting ambient light, this hurts after some time, and in the case of my watching a film, completely depleted my eyes of blood leaving white bags under my eyes
that is the hardware
as for the software if you read the comment i linked in the comments you are referring to i talk about an experiment i performed when i first received the oculus that shows that the brain error corrects the inconsistencies between the eye inputs at a pretty substantial distance, talking here of the blue and pink box experiment, in the experiment only the pink box moves, but when i showed it to a friend thae asked why when the two boxes began overlapping did the blue one start moving as well? i watching this all cloned on the screen in front of me explained that it still was the case that only the pink one moved and the blue one is being moved around by thaer brain, so if a simple experiment has my brain working overtime shifting still objects it would seem reasonable that a lot of the 'jitter' that is so often talked about when displaying entire worlds is your brain organising its inputs
as for what i believe is the right direction?
if we are talking about visual input manipulation.. i'd think projection, perhaps onto a dome or disc encapsulating the entire eye range including periphery of the eyes even with the pupil extended to the outer most regions of the eye
or one fantasy device i dreamed up used this recent research: http://newsoffice.mit.edu/2014/magnetic-hair-directs-water-f... ; if you watch the video, at the end, 57s in, you can see these hairs bending optics, what you'd do is have two rows of these hairs, one for each eye and each row bends to follow the movement of a pupil and each hair acts as a pixel
this way simulates two photons bouncing off a single object and reaching the eye, that would be wild
but mind that i referred to the effort as visual input manipulation, if you want to talk vr i think the only possible direction is bypassing our sensors and interacting directly with the processors, a la the suit in Zero Theorem : http://cdn-static.denofgeek.com/sites/denofgeek/files/styles...
Given tools to continually innovate and improve the digital world, I don't think I could ever grow bored of it. There would be endless permutations of possibility with which to express myself. I could make games, movies, physics, patch the stack. I could play God. Or I could let someone else play God and adventure through the worlds that they create. Then perhaps send a pull request.
An all-enveloping creative world sounds so much more exciting than the life I'm living right now. I could finally be anyone I wanted. Make anything I wanted. You might say I have this freedom now, but none of us really do. In a virtual world there is no central society to tell us how to conform and fit in, no traditional family structure. No tradition. Just feeding your short life's want for expression. I want that so much.
I can see so many people like me preferring a life like this. With this level of freedom. Imagine: Religion, money, and power could have little bearing. Meritocracy reigns, but even that could be ignored if you wanted.
Furthermore, I can see costs as becoming nearly negligible. No more consumption of material things beyond bare necessities: food, shelter, hygeine, and healthcare. Nobody needs a physical high rise view when the digital world can give that to anyone. I don't see why anyone else would need to buy the things we are trained to as consumers. (Would that be an economic problem? It very well could...) I don't see this necessarily ruling out a digital economy, but unless things are contracted and paid for on a per-demand basis, digital assets can easily be copied... an economic problem that might need to be considered.
If this were widely adopted (and barring societal collapse stemming from large swaths of society switching to a VR lifestyle), I can see it bringing ungodly amounts of funding to biochem research. We might elevate our concern for solving our biochemical longevity problems above all else.
Sorry for rambling and wasting your cognition with this idle pipe dream. I'm actually pretty pragmatic and don't expect for much of this to happen while I'm still young enough to enjoy it. Or at all.
VR is still missing the ability to simulate smells, tastes, and tons of other sensations. I think I would get bored in the world like that. Plus my real life is pretty fun.
It doesn't yet support "live programming" like this, but the hope is the framework would eventually be used to easily make these sorts of things.
So far, the focus has mostly been on the user input side. There is a constraint-based system for defining input values from different "devices", i.e. keyboard, mouse, gamepad, head mounted display motion tracking, arm motion tracking, etc. There is even a rudimentary gesture system, where things like a nodding gesture or head shaking gesture can be wired up and used to activate commands just as easily as a mouse click or key press.
The framework also supports multi-user environments, and I've already started building the text input controls (somewhat similar to what is shown here) to do things like this (right now, it's only useful for live chat). Eventually, I'll have buttons and text boxes and togglers, etc.
If you're interested in this sort of stuff, please consider joining me.
Join the mailing list: https://mail.mozilla.org/listinfo/web-vr-discuss
More info: http://blog.bitops.com/blog/2014/06/26/first-steps-for-vr-on... http://blog.tojicode.com/2014/07/bringing-vr-to-chrome.html
This is really good because I have a stomach condition that means being in a seated position is very painful for me, but using this I can stretch out on the bed, with a keyboard resting on me, and still code :)
So I'm actually pretty excited for something that needs me to look around all the time. I've still gotta try making my text editor twitch about all the time. A moving target makes more sense for our predator eyes, so maybe VR will finally allow people who work with computers not to have disproportionally more need for glasses.
My dream is a world where we eliminate physical displays from our computers altogether.
But we're definitely not there yet - it's going to 8, maybe 12K displays in a Rift to get the necessary resolution. But once it's viable, yeah, I hope to never buy another computer monitor again.
Then again, the idea of being cut off from the outside world with only your code to look at might be a less salient idea, so why not put a camera on this thing, then you could have "projected windows", basically like the holographic interface of tony stark without the holograms. I personally think that would be pretty cool.
Besides, I imagine that this setup would be cheaper than two monitors and a desk. I'd need to drastically improve my touch typing skills...sometimes even now I need to look down. Maybe in this set up there's a representation of my hands or better yet, a VR native way to interact with code that isn't a keyboard.
Addendum: I remember seeing something awhile back about building an IDE designed to be used with a gaming controller only...
> The matrix is an abstract representation of the relationship between data systems. Legitimate programmers jack into their employers' sector of the matrix and find themselves surrounded by bright geometries representing the corporate data.
> Towers and fields of it ranged in the colorless nonspace of the the simulation matrix, the electronic consensus-hallucination that facilitates the handling and exchange of massive quantities of data.
The hand-wavy "this might look like a book with a video in it and some neat drawings, but it actually contains a magical language that describes the world you're in" ...
... you're looking at a book written in a magical language that describes the world it's in.
But I'm not sure this is taking advantage of the true benefits that a VR environment brings.
What are the major benefits that a VR environment brings?
Screen real estate would be 1 benefit but I would have to see an implementation where this was done well before I could believe it.
What other benefits does VR have?
- Have a "keyboard cam" that only shows you the keyboard (and your hands when they're over it) displayed at some natural spot in the VR.
- Have a keyboard with capacitive sensing keys, reproduced in the VR with keys highlighted as a function of the proximity or contact with your fingers.
- Something like this [1]. Typing in thin air or on a table probably wouldn't work very well, but if the leap motion senses the keyboard as well as your hands, and the whole thing is reproduced in the VR, that might work.
[0] - http://www.emotiv.com/
Then I can legitimately wear an Oculus Rift at work!
I think what would work is some sort of hologram reality agumentation that is able to project 3d images in the environment you are in.
Again, this ergonomic aspect of the Oculus might be improved but my primary concern is the vision health aspect of it.
Its sort of like wondering about the health effects of wearing glasses.