Why VR/AR gets farther away as it comes into focus
matthewball.vc
matthewball.vc
“The assumption that the Metaverse is primarily an AR/VR thing isn’t crazy. In my book [Snow Crash, published in 1992] it’s all VR. And I worked for an AR company [Magic Leap]--one of several that are putting billions of dollars into building headsets. But I didn’t see video games coming when I wrote Snow Crash…
Thanks to games, billions of people are now comfortable navigating 3D environments on flat 2D screens. The UIs that they’ve mastered [keyboard and mouse for navigation and camera] are not what most science fiction writers would have predicted. But that’s how path dependency in tech works. We fluently navigate and interact with extremely rich 3D environments using keyboards that were designed for mechanical typewriters. It’s steampunk made real.
A Metaverse that left behind those users and the devs who build those experiences would be getting off on the wrong foot… My expectation is that a lot of Metaverse content will be built for screens (where the market is) while keeping options open for the future growth of affordable headsets.”
My personal belief is that the browser as we know it today will evolve to be more based around 3D, leveraging emerging technologies such as WebAssembly and WebGPU. As headsets become more widely adopted, WebXR is what will extend the web into VR.
There won't be a metaverse based around an app store model. It has to be open to everyone, built with standards and protocols in mind from the very start, and most importantly: no 30% fees from walled gardens.
I interacted with a lot of 3D content, games, apps when I was a child / teen but nowadays I largely prefer 2D for all my interactions with computers, be it games, softwares, etc. 3D just feels gimmicky and less productive than WIMP, especially on all the VR headsets I tried. I'm curious if other people feel different.
for most of the games and other VR entertainment stuff, this is precisely what the producers what of you. they throw around terms like immersive. they want you in the world they created and believe that it make the experience that much more when you forget about where you really are and believer yourself to be in the virtual place. there's a definite time and place for that idea, but have never once felt that the content i was "immersed" in really had me at a level of suspension of disbelief to the level the producers had hoped.
at that point, what is the point of it. i am not one bit intrigued by theZuck's vision. i do not want to hang out in a virtual conference room around a virtual conference table looking at cartoon versions of people because it's cute. you can cute the fuck out of it until the knob goes to eleven, but it doesn't make this boring as meeting any more enjoyable.
I think you might be conflating 3D graphics engines that render a 2D image on a 2D monitor with modern stereoscopic 3D headsets.
Also, the Quest 2 is kind of old now, newer headsets are vastly superior with regards to blur and streaks.
But wow who cares, when I'm playing a third-party VR mod of half-life 2 from 2004, it seems so real. Despite the low-poly meshes and blocky textures that come with games of that era. Blown away by the feeling of actually being there. I get vertigo balancing on the de support beams of the huge bridge I have to climb, disgusted at the howling zombies attacking me while I hurriedly try to stuff shells into my shotgun.
Those too-big and underwhelming pixels? I really don't have time to notice them.
I bought this headset for only $50 more than the $300 you mention (yes now it costs more). For desktop work yes I prefer my $220 4K monitor hands down. But wow this is something else and it's only getting better.
Edit: Sorry. You deserve a serious answer. I'm old and not someone who wants to walk around and spin with a headset on. I'm someone who wants to use something similar to a flight simulator or Google Earth, so I put a premium on the niceness of the view.
But the things you mention benefit the most from VR. Graphical resolution is indeed a bit lower with Flight Simulation but the feeling of flying is much more accurate. I've flown real Cessna 172s and I've never really felt like flying in a PC simulator until VR came around.
Google Earth is also quite amazing in VR. Of course for both experiences you need a serious PC, neither flight simulator nor google earth will run on the headset natively.
The resolution and colour depth is really one of the least important things in my opinion (and they will be solved in due course, the first TFT screens were also quite horrible and still we played sims on them).
I would be happier with VR that was all done on a high powered desktop computer where the headset is just two high quality screens tethered to the desktop computer. I see the problem as trying to put all the processing power in the glasses so people can move around. I've done Sandbox VR here in the Bay Area where I can walk and spin and it's amazing. I'm comparing Oculus to that and feel Oculus comes up short because it's a home consumer setup that has to be affordable.
I'm going to shop for the type of VR I want. I realized in this thread that that's what I need.
https://www.urbandictionary.com/define.php?term=no%20offense
> It lags and jumps bit and does not scroll smoothly as I look around.
That hasn't been my experience at all, not since the DK2 I originally had, maybe on the google cardboard, I never tried that. A headset with dedicated tracking works amazingly well in terms of your rotational/positional view of the world.
https://en.wikipedia.org/wiki/Open_Cobalt https://en.wikipedia.org/wiki/Croquet_OS
With Croquet, you could literally jump from, say, your favorite 3D game into a virtual business meeting, controlling your same avatar, and without either 3D space being hosted by the same entity--directly or indirectly. It's the embodiment of some of the most fanciful visions of a Metaverse, and it has existed for quite some time.
But Croquet hasn't taken off, despite intensive efforts, which tells me that not only will our 2D interfaces continue to stick around, the 2D digital world is likely to remain overwhelmingly predominate for quite some time still. At least outside isolated islands.
And in any event, I don't see how that could change unless and until a fundamentally open, interconnected, and extensible platform like Croquet sees adoption. A world of closed gardens could never be as immersive and expansive. As large as Facebook has become (and as AOL before it), it simply doesn't and couldn't have the same underlying impact as the WWW collectively.
[1] NB: the Wikipedia article for Croquet_project now forwards to Croquet_OS, a commercial project. I hope Croquet OS finds success. I can understand the thinking that on the modern internet new technologies necessarily require commercial backing. But ultimately I think that's a reflection of some fundamental dilemmas on the modern internet regarding the uptake of new technologies; a dilemma that likely has no real solution unless or until something else comes out of left field to toss the ecosystem into disarray. That something is unlikely to be the Metaverse because otherwise Croquet or something substantially similar would have already taken off long ago.
https://wordsrated.com/amazon-kindle-e-book-and-kindle-unlim...
But those are fairly new, so take the last generation. The PS4 sold about 100 million, and Xbox One about 50 million: https://www.theshortcut.com/p/how-many-xbox-one-consoles-mic...
So the leading game consoles outpace the leading ebook reader, but not by all that much. There are other gaming platforms, but there are also other reading platforms. Some people read kindle books on their phones, kindle books are 72% of the ebook market, and ebooks overall are a third of the book market.
~825 million printed books were sold in 2021. Reading is such a common pastime that people just don't talk about it. It's also not a particularly social activity unlike most video games so most people won't bring it up unless they read a very popular book that you might be aware of.
If you want to make VR "work", you have to also have non-Euclidean Geometry. Things need to be smaller on the outside. You need to be able to affect the "world" through intention, not action. In a word, you need magic. Which dovetails pretty well with video games.
This is why a 'modern' desktop still mimicks a messy desk from a 70s office worker. The very easiest computers skipped that step due to technical limitations but the mainstream users had a hard time grokking them. Only with the mobile phone have we started to really leave that behind. VR will be similarly rooted in the past until people find their footing and embrace new ways to interact.
And why meeting rooms in VR have to have chairs, tables and people have to draw on virtual whiteboards instead of just in the space in front of them in 3D. The tech is getting there, but our minds take even more time.
They are familiar sure, but who wants to put on a headset for a meeting….
I feel like being familiar here is also being uninteresting.
Feels pretty damn close to the real thing, close enough I think skills gained there would translate to the physical game. And you don't need good weather, have to travel, or have to wait for other players to play the mini golf.
Also being a video game allows Walkabout to do some courses that just wouldn't be feasible (not physically possible or too expensive to build) in real life, like the Myst puzzle one, or that one hole that's based on M.C. Escher's Relativity that alters gravity depending on where your ball is at.
Still need to play them multiplayer, but I want to. I watched a game played in VR which was basically an interview with the creator of Myst and that was a lot of fun[1]. I think this is what you're referring to, meetings being more fun over a game of mini-golf than just a Zoom call.
This is VR’s biggest problem. Normal people aren’t willing to put on a bucket on their face, even if it represents the future. Instead most people would rather form all kinds of weird and untrue speculation. Case in point: “I don't need a VR library full of fake volumes of Encyclopedia Brittanica, because I already have Wikipedia”
No one has actually built a library in VR because the screen resolution is too poor for reading large amounts of text. All you’ve proved is that you still haven’t tried modern VR.
It is not unreasonable to fear that personal AR isn’t going to get popular until goggles are comparable in size to regular glasses.
They're not all that far: https://www.engadget.com/2019-01-03-vuzix-blade-hands-on.htm...
Convenience is contextual and no doubt the way 2D WIMP screens have been built work and will keep working, but nesting that in a 3D experience makes new room for new feature interactions to complement the same datasets with more interface layers doing things you just can't do with real world physics.
The trying to create reality is simply recreating familiar paradigms to help new users get a foothold. You can't jump right into the most modern user interface. Even for advanced users.
This is why a 'modern' desktop still mimicks a messy desk from a 70s office worker. The very earliest computers skipped that step due to technical limitations but the mainstream users had a hard time grokking them. Only with the mobile phone have we started to really leave that behind. VR will be similarly rooted in the past until people find their footing and embrace new ways to interact.
And why meeting rooms in VR have to have chairs, tables and people have to draw on virtual whiteboards instead of just in the space in front of them in 3D. The tech is getting there, but our minds take even more time.
I couldn’t help but think that I wanted neither and the folks pushing this idea were people who wanted conference rooms or stadiums of people watching them, but I’m not sure anyone else would be interested.
But this is also similar to the idea that some people want their MMORPGs to have a single, non-sharded instance, where everybody can play at once, which sounds hellish to me TBH. Millions of people all competing for the same items, quests, etc. If you were offered a VR version of Disneyland, which would you prefer? One where there were very few people and you could walk more or less directly onto the rides? Or one where the streets are packed and the queue times are multiple-hours long?
The rush to reproduce these less desirable experiences in VR is utterly depressing, and for that matter the descriptions of The Street in Snow Crash didn't sound that appealing to me either.
You could have a row of clubs on The Street that are proportional to each other but not to their interiors. All clubs are 5 meters wide. Large clubs are 7 meters wide, and massive clubs go to 10. You can look at the outside and predict what's going on beneath without walking by a SuperWalmart.
And then I believe it's in Neverwhere (?) where parts of the spaces of the Underground are borrowed from other places. So the average density is always the same but pockets can still be made.
If you're worried about how to handle physics, a really simple solution here would be to slice your VR onto multiple planes of existence, from outside to inside. You have a virtual street full of fifty businesses, you can lay out the insides of the business into a Euclidean space, and then doorways are wormholes from a less dense plane to a more dense one, so you can walk around the block faster than you can walk through one of the buildings. Recurse that for as many layers of tesseract as you want to implement (why not rooms in the building that are bigger than the inside of the building too?). But once you've broken 1:1 people are going to want to change their minds, so you might be better off using some sort of graph to represent space instead of a 3d matrix.
We're currently seeing the tail end of progress in transistor technology, evidenced by chip fabs now costing 10s of billions of dollars each, smaller improvements between transistor nodes, and so on. This is a huge obstacle and something which these AR/VR companies just have to hope and pray TSMC will indefinitely, magically overcome.
At the same time, consumer expectations are continuing to increase year after year. We're so picky now about screen pixel densities, color gamut, contrast, refresh rate, etc compared to 10 or even 5 years ago. No doubt in 5 years our expectations will be much higher.
It’s never wise to say something is impossible unless it breaks laws of physics. Computing wise, it’s already possible to stream a PC wirelessly to a VR headset either right now from a local network or even the cloud in the very near future. I mean we’re already streaming AAA Xbox games to phones now.
Modern AAA games on a 32" 4k monitor that I'm sitting right in front of are pretty damn beautiful and immersive.
By comparison, anything in VR is low res and low poly.
Of course this is a natural effect of the technical constraints* but the 3d and head-tracking aspects of VR seem less important to me than the crispness of a screen.
* I can't find a plausible explanation for the absurdity of the graphics of (e.g.) Meta which despite an apparently unlimited budget can't muster graphics exceeding what I experienced on a 1024x768 on my Pentium II in highschool.
But it's just one app on a quest, I uninstalled it and carried on using it for what it's meant for (gaming), rather than Zuckerbergs failed attempt at forcefully making it into some new market.
> They're basically just an android phone built into a headset, and they're capable of significantly less than a PC.
You're missing the value in these things, it's also capable of significantly more, because it's not all about how fast your GPU and CPU is. A lot of work has gone into taking commodity mobile technology and tightly integrating all of the sensors in a way which is low latency enough to make VR a pleasant experience rather than a horrifically sickening one.
Apparently, Facebook’s marketing is so good that they’ve convinced most of the world, even techies, that they own and control the VR AR equivalent of the internet.
> Metaverse is literally just AR and VR over the internet
Not really, that's way to vague, it's come to mean something more specific. VR games existed first on VR and they are just games, often over the internet, they are much like games on PCs and Consoles but with an extended human interface and perception.
That’s my point. meta Horizon is not the entire metaverse. It wasnt even first, nor will it be the last destination on the metaverse
> Not really, that's way to vague, it's come to mean something more specific.
Actually, my description is straightforward and to the point. It’s your description of the metaverse that is vague.
It's the first consumer level headset at an acceptable cost/performance point. Nothing about it is particularly good, merely acceptable. And I'm pretty sure they're being sold below cost, FB is subsidizing them, hoping to sell you more ads in VR.
On the software side, is there a comparable standalone game library to the Quest Store and App Lab?
Yes and this is the key part, it's the first affordable option. Also if you combine it with a PC then it's both affordable and decent quality
> plus some clever position tracking software
I don't get why everyone devalues this so much, the cost of new integration for something like this is high, once it's been done it's easier for others to follow, but how we got here wasn't just people throwing expensive hardware at the problem, it needs much more than that.
> I'm pretty sure they're being sold below cost, FB is subsidizing them, hoping to sell you more ads in VR.
They were selling it bellow cost, now they've added 100 $/£ to the price tag to put it above cost. Which was one of the last thing Carmack got them to do before he left. He didn't want the incentives to be messed up since a large free/sideloaded games ecosystem has emerged and it doesn't make sense to be at odds with that.
You can have all those fancy graphics right now if you want them on a Quest2 or Vibe or whatever at almost 4k something resolution depending on how you measure it. You just hook it up to a PC and play a VR compatible game on Steam, then you get the power of whatever PC GPU you have. You are looking at the games that run standalone on the Quest2, they are stuck with a massively underclocked mobile GPU to make sure your head doesn't melt.
Not all games are about graphics, and I find the most fun and replayable ones certainly aren't. That's where the standalone headset games have their place... The killer feature of these and the Quest2 in particular is that you just throw the headset on and pick up the controllers, it's like the gameboy of VR, no need to hook up external sensors, a gaming rig and a tether, instead you can stick it in your bag and use it almost anywhere.
It depends what you want out of gaming, I haven't owned a console or "gaming PC" in 20 years, I was never a fan of the "wii" type games, I'm an old skool Doom and Quake guy, so I wasn't convinced about VR until I tried it, and yet now i'm one of those weirdos playing beat saber on expert. I don't care much for modern AAA graphics so their absence doesn't really bother me, I grew up seeing the incremental improvements and it's just more of the same to me, most AAA games all appear to be a small variation on the same mechanic, nothing much changes there. What interests me is novel game mechanics with replay value, graphics just need to be acceptable for that. I'm not after a "visual experience", that gets old real quick. But it's all subjective... just don't knock it until you give it a proper try.
A stand alone Meta Quest 2 runs on an ARM processor that’s weaker than Samsung’s flagship phones which are themselves weaker than iPhones.
> By comparison, anything in VR is low res and low poly.
That’s because you’ve completely ignored PCVR. VR visual fidelity is pretty good when it’s running on a PC that’s capable of VR. Just use VR headset, even the Quest 2, with a PC. SteamVR is 6 years old now.
For me the killer use case would be to use VR as a virtual desk, just create an infinite amount of windows/Linux displays, not some kind of crappy mobile browser like oculus does, though lenses and displays need to improve a lot to be clear like a normal multi display setup.
That leads to several hard problems: you don’t just need to generate a high-quality scene, you need to render it twice because your brain is used to the distance between your eyes (16K==8K per eye) and because your position isn’t static it needs to update very quickly when you move because your brain is very used to how that works in real life. You have a budget measured in single digit milliseconds to that or a fair number of people will get motion sickness and many more will notice the lag and find it detracts from their experience.
Here’s a good overview:
https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
That's not how Ks work.
And your explanation doesn't cover why the minimum number of cameras would be 12.
> And your explanation doesn't cover why the minimum number of cameras would be 12.
This is covered in the article. The key thing to understand is that there are multiple tasks using cameras: AR needs to track the world around you, you need eye tracking to handle saccades and conscious direction, and tracking hands, arms, legs, facial expressions, and body position accurately requires multiple cameras with enough parallax to build 3-D models which are accurate enough to generate realistic, stable imagery. People will really notice if things bounce around or deform or if you punt on some of the proven and end up with Meta’s leg problem.
The key point to take away here is that AR/VR aren’t one hard problem but a collection of hard problems which all need to be solved for most people to find the experience compelling — all with tight physical constraints.
Nothing explains why 12 specifically. And if 12 is a minimum, what's the optimum?
> The key point to take away here is that AR/VR aren’t one hard problem but a collection of hard problems which all need to be solved for most people to find the experience compelling — all with tight physical constraints.
And it's a pretty bad point because it only works if you constrain yourself to head-mounted (including battery) standalone devices. The article gives a nod to other ways of doing it, but otherwise ignores them for all its main arguments.
And you don't need optimal to be compelling.
The article isn’t a complete survey of the entire field, it’s talking specifically about the most popular approaches and extrapolating what the author believes a breakthrough popularity device will look like. It does mention other options such as Apple’s rumored support devices or how industrial applications can take advantage of their environment. The challenge here is price: these systems are expensive and getting people to buy them means they either need to be cheap (impossible) or give a lot of value, and companies are assuming that means something you can easily move around in and use for long periods of time (not many people are going to set up a VR room, buy a headset and sensor array, and a pricey PC).
While it’s true that it doesn’t need to be perfect, some of the hard problems spoil the illusion - people get sick or suspension of disbelief fails - and it has to be enough better than the widespread mainstream entertainment systems to justify the much greater cost. Currently we haven’t hit the point where many people, even many self-described fans, use these systems heavily and that makes it an expensive hobby.
How many minutes per day would you consider “heavy usage” given the current entertainment/gaming focus?
This ties back to the price question: this currently costs noticeably more than console or even PC gaming, both of which are more social. A mainstream product needs to offer something you don’t get there to justify the extra money.
But I think you’ll find that your priors are quite a bit off at least in terms of “heavy usage” of VR though. Without any snark, I highly suggest talking to people who work in the industry if you’re curious.
Note that a new Quest 2 costs $400 with the price hike (previously $300, and sometimes as low as $250 when on sale). Compare that to $500 for a PS5 or Xbox Series X. I don’t think it’s correct to say that VR “costs noticeably more” or doesn’t have something extra to offer; don’t discount the novelty factor of VR to move sales.
The Quest 2 is reasonably priced, yes, but every review mentioned drawbacks (often due to Facebook rather than the concept) so the question of satisfaction per dollar is still pertinent. The people I know who are VR fans all use PCs which is where my cost priors come from - high-end gaming system rather than console level - but hopefully that’s shifting. I was definitely expecting this to be more noticeably popular by 2023.
Assuming inside out tracking, redundancy is required, since you can make the majority of the cameras useless by standing in the corner of a room with white walls. Features are required for tracking.
Maybe the future is LiDAR, or something new, or old (military gets along with IMUs).
Well, we're talking about the future. The Quest 2 implementation proved that the future is definitely not lighthouses [1], especially for mobile, which is the neccesary future of AR. Lighthouses are a technology from a time when there were no SOCs with the hardware required to support the cameras. In the end, we'll probably end up with a mix of SLAM cameras, and lidar.
Full body tracking is all that's keeping lighthouses relevant. Maybe that goes away with more cameras. :) If you have enough cameras/lidar, you get tracking of all world objects, not just those with sensors attached.
1. https://dl.acm.org/doi/fullHtml/10.1145/3463914.3463921
> The results reveal a significantly higher accuracy for the Oculus Quest 2 compared to SteamVR Tracking in the height of a tracked object. Furthermore, the Oculus Quest 2 tracks its position with substantially higher precision than SteamVR Tracking.
Only if they're cheap enough, but if you're in the situation of "cameras are so cheap we'll put in 40" then the idea of "minimum 12 cameras" is extremely not an issue.
Just make bridge cooperation between touch device, laptop, and display/XR with faster nearfield links to remove HDMI and such. And make it a standardized open ecosystem that casts between each.
Quest has all of the right sensors, but it locks away too much expensive computing power for a single dedicated purpose. It's extremely convenient but it's also another device to MDM and separate from a users other apps.
He's really echoing John Carmack, who said, around the time he quit Oculus/Facebook/Meta, that the headgear had to get down to swim goggle size to get any traction, and eyeglass size to go mainstream. You can just about do that now if you're willing to have a wire connecting you to the game console. Sony is doing that with their "VR2" headset. It connects to a Playstation 5.[1]
But standalone, wear all the time is still a ways off. A lot of hardware has to be crammed into the headgear.
I'm also pretty excited about the next generation Google Glass with real time language translation. That seems like it would be awesome for international travel.
I have no real interest in it for gaming though. And VR is unappealing compared to AR.
This should be the first entry in How To Get Mugged 101
Then there's the overall likelihood of getting mugged in a given place. I'm not too worried about getting mugged in Japan. And I'm not interested in going to high crime places.
Let’s see if Apple is able to change your mind this year if they release their MR headset. Unlike Meta, Apple historically focuses on applications first instead of games.
Apart from that, AR has even more hurdles to solve than VR, especially cost. To give you an idea, Apple’s upcoming $3000 MR VR headset is “cheap” compared to its delayed AR headset.
https://www.bloomberg.com/news/articles/2023-01-23/apple-rea...
I think this is going to be a huge driver of VR adoption, along with travelogues.
The reality is that now everyone can have their own private movie theater to watch sports games or whatever they want, if they have a headset.
It’s not much a stretch to imagine limited access events where they mimic a stadium with assigned sections you can’t stray from.
So in conclusion, yes, a better world is possible. It’s just that it’s intentionally sabotaged.
What's wrong with mounting the supercomputer on the ceiling and having a telephone cord run from the glasses to the supercomputer?
Adding in fiddling with a cord will make me just slightly less likely to say: sure, it's worth the effort to get into VR right now. I really want a full body tracking setup, but if I have to wear special devices for it, then it's a no go. I don't mind putting up a bunch of cameras though.
Another issue is rotation, many people have a bias to the direction they turn, this twists the cord, AFAIK we aren't even close to having the tech to spin a high bandwidth (multi-gigabit) connection, I suspect that it may be possible to do with fibre optics.
But AFAIK most designs are looking at doing hip mounted or wireless supercomputers. We already have backpack supercomputers for VR.
Maybe this will change with the upcoming $3000 Apple headset?
I know SimulaVR’s upcoming headset was designed for work from the start.