Nvidia CEO: We'll be solving VR immersion problems for the next 20 years
trustedreviews.com
trustedreviews.com
And, I'd like to imagine he has some enjoyment at the thought -- if you saw his keynote showing off a precomputed(ish) raytraced walkthrough of their new office space, he was really enjoying himself and the possibilities.
What Hsun has is an incredibly good window into the silicon side of the problem, and he knows how many chip cycles he's got to address the amount of computation needed for physics, visual realism and presumably a latency budget to allow wireless support. Plus he can estimate what screen tech will be doing.
It's a prodigious amount of R&D, and it is a bit breathtaking. Finally having a reason to do it, and a taste of how appealing it can be is pretty cool.
Of course, 3d chips look fairly achievable and might eke out a few more cycles, but things will get interesting sometime before that 20 year deadline.
14nm FinFet GPU's are only coming in now, and they'll stay on 14nm for at least 3-4 years. 10nm GPU's will probably come out in 2019-2020, 7nm GPU's will come out in 2025 or so, and 5nm GPU's will come out in 2030, that's already ~15 years and considering that everyone is predicting that we'll have to stick to processes longer as the small we get the more time it takes to actually optimize them for good yields and performance you probably have 3-5 years there too.
So even if we do not adopt any major process changes like switching to graphene, solid state photonics to replace the metal pathways and even optical transistors or w/e else people are cooking up to enable us to continue to improve our integrated circuits he can still be correct about the time frame.
Also regardless of what happens or not he was talking about true VR immersion this means that allot has to change other than just how good and convincing the graphics are this also includes learning to develop VR applications and as VR is a very new media that alone can take 20 years.
I think a lot of VR's biggest hurdles for that time aren't dependent on processing power. Focus and resolution issues will side-step some of it (seriously, our eyes are terrible outside of their central focus region!) because there's simply no need to render everything flawlessly as on a TV. Beyond that, there are going to be all kinds of design hurdles. Right now, we have nausea-inducing roller coaster rides and people running into walls because we can't sync up the (already impressive) visual experience with inner-ear confusion and physical boundaries. Just working out what the most compelling designs are for our current VR tech is the work of years, and that's without doing it on the treadmill of ever-improving tech!
Yes, but we are just starting 10nm now. Go two more nodes (if I understand correctly that is meant here by cycles) and you are at 5nm.
https://en.wikipedia.org/wiki/5_nanometer
to put that into perspective: "An Australian team announced that they fabricated a single functional transistor out of 7 atoms that measured 4 nm in length."
Maybe we can make our way down to single atom transistors eventually, but who knows how long it will take to make that commercially viable.
Some volumetric/panoptic displays are getting closer, along with eye tracking, but this seems like one of those things you cannot fix in software. Just adding higher pixel resolution, improving "physics", or creating a more beautiful environment will not solve this. So, 20 years is probably right.
Alternatively, they could use a modified biology, where the direction the head is pointing chooses the relative DOF, basically you render a fairly wide depth of field around whatever the head is pointed at.
Anyway, you can get a pretty lifelike experience with just some basic atmospheric perspective. Look at photography with very wide depth of field [1], so everything is in focus simultaneously. You sorta just get used to it, and accept it.
I think what we really need is a way to target specific focal distances via some sort of dynamic refraction. Otherwise we'll never get the proper motor feedback needed for immersion.
Source? I'm under the impression that focus is a brain-first, muscles-second process. If the object was already in focus, I don't think your eyes would mind.
As I understand it (which I may not) -- this paper (which purports to be the only paper to have tested this effect), says that they test divergence by having the images displayed at various distances to replicate varying levels of divergence.
They see that the closer it is to a realistic vergence/accommodation match the more comfortable the eyes.
However -- it should obviously be easier to tune a 3D image which is at a realistic distance than one which is not.
It seems to me, they may very well just be showing that effect, just showing that it is easier to adjust the illusion to render realistically at distance. This would make perfect sense.
This would then just argue for better tuning.
https://research.nvidia.com/publication/near-eye-light-field... https://www.technologyreview.com/s/534971/magic-leap/
But instead of trying to adjust for how our brain works, we could give an enhanced experience, so that IRL will feel "clunky" in comparison.
Perfect photorealism is fairly worthless without a solid method of interaction and the ability to literally suspend disbelief.
Direct neural interfaces will be capable of everything, not just rendering.
That said, I can't help but think of a scenario where we have functional neural interfaces but haven't yet mastered the human brain's grammar, and thus resort to "driving" the brain with traditionally-rendered images rather than feeding it the information required to create a scene from scratch.
In such a scenario, the requisite quality of the traditionally rendered images might be interesting. Would photorealism be required, or simply a low-quality rendering to convey general structure? Presumably the brain would enhance the latter case to the point of photorealism—or at least to the dream equivalent.
Rudimentary technology that does just this has been around for a while as assistive devices for blind people [0] albeit only rudimentary shapes and movement can be portrayed.
I believe ultimately the most effective technology will be that which integrates directly with the visual cortex, which I read somewhere has neurons arranged somewhat isomorphically with the visual field [1]
I think this should be sufficient to stimulate both higher cognitive and the more reflexive automatic visual systems [2] though IANAN and really haven't look at this area in depth for a number of years.
To get into the realms of pure science fiction about it [3], this could even be done by way of the pineal gland, which has some optical features that are largely unused. Perhaps it was put there like that for a reason ...
[0] http://www.fightingblindness.ie/cure/retinal-implant-technol...
[1] https://en.wikipedia.org/wiki/Retinotopy
https://en.wikipedia.org/wiki/Visual_prosthesis#Dobelle_Eye
It had bad resolution, required those people to learn how to see using it, and had biological problems (wires exiting the skull); all these do not seem to be fundamental problems, but ones that require incremental development.
What about dreams?
At first I thought the idea was pretty ridiculous, but now that I've given it a bit more thought, I'm beginning to think having to carry around a backpack like this might not be such a bad compromise between the traditional stationary wired VR setup and the holy grail of fully wireless VR.
Carmack was even saying stuff like you need a 12k or 16k display to have the "ideal" resolution in VR, years ago. Well, we're not going to get those kinds of displays in VR headsets anytime soon, and even if we do, that's orders of magnitude more performance that's needed compared to what we're using in today's PCs.
On top of that, you also need "virtual reality" to be as close as possible to actual reality in terms of graphics fidelity, so that's a few orders of magnitude more in terms of performance needed. If you want "Matrix-style VR", then yeah, that won't be achieved for at least another 20 years. The headsets will also have to become "glasses-like" to increase adoption and make them more convenient, which is going to take at least another decade, too.
All of that said, while I do think the 1440p resolution is way too low, and probably even 4k won't be the "optimal" resolution for VR, I do think VR will start becoming "mainstream" within the next 5 years. I think many took his comments to mean that VR won't be mainstream for another 20 years, but that's not what he was saying.
[1] Including my partner who gets nauseous from even short car rides, to say nothing of boats, airplanes, or the DK1.
Reductions to this latency is an ongoing effort and companies like Nvidia are definitely in that space.
Socially/culturally yet another thing separating hard core gamers from the general population is probably not good. On the other hand, any future hard core VR gamer being able to jump on a real life boat and not puke is probably very good for the boating sports.
Another interesting cultural point is like many other people I don't get seasickness nausea even under pretty bad sea states until someone else pukes, then I start feeling queasy. I suspect this will be an issue for VR gamers at LAN parties. The first guy to puke at a LAN party who makes everyone else queasy is going to be a meme in 2030. I would imagine playing sound drops of people vomiting over teamspeak will be considered VR cheating, and likewise a soundboard app to generate vomiting sounds will be a cheap source of money.
Much as energy drinks are identified with hard core gamers, its not unlikely scopolamine sea sickness patches will be a thing in the gaming scene. Along with fake herbal preparations that don't work, people hiding the patch to pretend they're elite and seasickness proof, etc. And of course the patch has some interesting nasty side effects which will likely become part of gamer culture. I predict scopolamine patches will be sold decorated up like energy drinks.
Anyway if you're looking for some way to "tag along" with VR as a startup, look into anti-seasickness technology.
I don't know how it would play out, but speculating, you might render a low-poly monochromatic "skeleton" scene and then simply suggest to the brain ways to flesh it out with feeling and texture. Show the user photos of war-torn Europe in 1947 beforehand for example. Even if it means direct neural interfaces, considering that we're 20 years out using the current approach, maybe that isn't so far fetched.
To be economically viable, the tech needs to be sold to be mass-sold directly to consumers. So it needs to be mostly safe, even when it has bugs, and usable by anyone, without months of "training your interface device and brain".
Even if there was some progress with "direct neural interfaces", without a breakthrough in nanotechnology I'd be we're probably more than 20 y away from something usable outside of a hospital or a military lab that has a different cost/risk/benefit equation from the "average joe" consumer...
And you probably don't realize it, but to get that "spoofing" part right, you might have MUCH HARDER problems to solve than having a ultra-high-re ultra-low-power ultra-low-latency mobile VR set...
Or visual systems are too tightly integrated with our motion systems and other sensory systems to be handled independently, just as traffic lanes are too exposed to other influences (unpredictable people and things) to allow vehicular autonomy.
In both cases we want to see the thing we want to do by itself but there are deep interdependencies.
Has Nvidia had any success at all in mobile? At least Intel has managed to get some Atom chips in some obscure Dell tablets and Asus mobile phones.
I'm very curious about this - Nvidia's whole mobile business looks like a flop and Huang's circumspect dismissal of it sounds like sour grapes to me.
[1] http://wccftech.com/nvidia-tegra-x1-benchmarks-apples-a8x/
[2] http://marketrealist.com/2015/03/why-nvidia-continues-to-foc...
[3] http://www.extremetech.com/computing/227816-how-intel-lost-t...
Delivering good chips for mobile use? Yes. Tegra chips have been showing good results in benchmarks.
But making money out of them? Not so much. The whole mobile chip industry became a race to the bottom (with prices) many years ago and most companies in the business are struggling as Samsung and Apple make their own chips and the rest of the market isn't worth a lot of money.
In recent years, Google Pixel C, Nexus 9, Nexus 7 (2012), that Project Tango tablet, and Nvidia's own Shield devices (TV, tablets, handheld). See: https://en.wikipedia.org/wiki/Tegra
Not the best-selling lineup, but a fair amount of developer mind-share thanks to Google's penchant for Nvidia SoCs and Nvidia's own outreach.
Unless they invent/integrate tech like Project Tango into most phones and make it work for spatial location this is not a problem that can really be solved on mobile.
• The resolution has to be a lot higher.
• The physical worlds do not behave according to the laws of physics.
• The environment you’re in isn’t beautiful enough.
Those are not game killers. If points like that were game killers, we wouldn't have had the Atari or the Nes.
This is where I feel most people in the VR space agree that "VR arcades" will become a lucrative industry.
If you're doing 1:1 comparisons, then it should be about 10 years behind VR because you have to transpose the same world that you do in VR but over the real world.
“First of all, VR displays are a little too cumbersome. It has to be much more elegant, being connected by a wire has to be solved. The resolution has to be a lot higher. The physical worlds do not behave according to the laws of physics. The environment you’re in isn’t beautiful enough. We’re going to be solving this problem for the next 20 years.”
"Solving over 20 years" / "20 years from being solved" are very different things.
How do you suppose that will happen? On-device computing hardware or a wireless tether?
The latter appears to only be maybe possible with microwave radio, but requires constant line of sight which is a huge issue.
The former may be possible depending on your definition of hidef. IMO I doubt it though, much of what makes VR immersive is having a HMD lightweight/comfortable enough that you forget it's on. That's hard enough when the only hardware in the HMD is essentially screens and an IMU. I can't imagine the added weight of CPU/GPU/batteries/etc not making a significant enough impact on comfort to make it not worth it, in the next 20-36 months.
For reference, my standard of hidef VR right now, not in 20-36 months, is Vive level resolution/FOV/tracking accuracy and latency; I wouldn't put GearVR in that group, or any platform without submillimeter positional tracking for that matter.
about the VR industry from the top comment of that page
Edit: Thanks for the downvote
Sensory augmentation will probably start with dream augmenting tech (headsets that are able to give you "good" dreams by learning your neural firing patterns), then as delivery systems, nanotech, processing and brain knowledge gets better, we'll see it evolve into waking experience augmentation.
Yeah that's not what anyone here is talking about.
My point was those problems might be on the horizon of nvidia in the next 20 years, but its not even scratching the surface of what VR needs to be.
But its more reasonable to think well at least get a near-perfect audiovisual virtual reality headset in 20 years. If the numbers you're throwing around here are accurate, then there will be an 80 year stagnation in vr immersiveness, when the headsets are great but anything motion related feels fake, but computer/neural interfaces still limited to prosthetics, or at least out of reach for consumers and not powerful enough for vr.
80 years is a lot longer than TVs have been around, there's plenty of time for graphics card companies to rule the world.
The first generation VR also faced the same liability issues. Specifically, the Atari Jaguar VR was cancelled because of this.
VR will never make it as a consumer product.
Ever.
There was a ton of innovation. But frame rates, even on half-million dollar hardware, had to be in the 30hz range. Knowing what we know now, there was no way for that technology to take off, even for those with very strong stomachs.
The minimal technology package to do a good job has only very recently been available for almost any money. I think it's too early to call it as a 'never'.
I used those early systems, definitely ones better than the Jaguar at least, and in the same year it was released. And, I'm saying anyone who thinks the Jaguar's failure means modern VR is doomed is missing some super basic things about the threshold at which VR is compelling.
And, I'm saying we are just over that threshold, sometimes, with some experiences, and that there was never a snowball's chance in hell that any kind of compelling VR experience could have been had in the early '90s.
If the Jaguar had been compelling, the liability issues would have been sorted, full stop.
Anecdotally, I've found even my 70+ year old father-in-law responds well to Chaperone bounds in the Vive, he 'gets' them immediately. A friend who is partially disabled is also successful at using the Vive, despite some mobility issues. None of that (or the attendant safety) matters, though, if the tech isn't compelling.
Take your uninformed nonsense elsewhere.
Are you implying it doesn't happen?
Also, how do you make something like Call of Duty when you're limited in your movement by walls?
And you don't make Call of Duty for VR. VR experiences have limited movement, or you use teleportation to move around large spaces.
And with restricted movement, VR's use case becomes much more limited.
Face it - VR is a fundamentally flawed design.
It's one of those truly terrible tech ideas that needs to go away.
(AR on the other hand, does have potential...)
You're either an idiot or a troll.
If you want to defend this horribly dangerous product, feel free. But normal people will continue to call out such an awful product design.
Really, we need to get rid of VR.
You don't. Many existing game genres don't map well to VR in a direct port of existing mechanics.
From a game developer's perspective that is one thing that makes it exciting. It's a blue ocean of new design possibilities. I'm an optimistic person but I believe we'll find new designs that are equally compelling to the most popular games on 2d screens. Different games for different platforms.
> VR will never make it as a consumer product. > > Ever.
Which it seems you are not qualified to give expert advice on. You might be right but you might get less downvotes if you adjust your tone a bit; Posting a single video and the jumping right to this conclusion also comes off as a bit lazy or something :-)
Also, the downvotes are a natural result of all the people on this site that are invested in VR and want to see it succeed, so it's expected.
Yep, but predicting the future of VR citing only a single video doesn't come off as very convincing.
I don't think liability is going to be a big problem though. People were destroying TVs with Wiis the whole time when it came out, but the Wii was still a huge consumer success.
I would say the bigger problems are to do with the fact that for room scale VR you need a dedicated space which lots of people simply don't have the room for. The aspect of disconnection from your environment is a problem, but I believe it's mainly how it affects social interaction that is problematic rather than issues to do with colliding with things. The chaperone system in the Vive is pretty good, and if you're playing a seated game in the Rift, you're not crashing into things anyway.
Personally I think that the most sensible arrangement for VR is coworking spaces with large fully tracked VR areas for groups and only enthusiasts would have them at home, but my understanding is that Vive and Rift are selling pretty well, so it's quite possible that I'm wrong on that.