Meta Quest Pro – Bad AR Passthrough
kguttag.com
kguttag.com
I am fairly confused by the low res cameras used in the QPro. I have to assume Meta were just heavily constrained by the need for them to do double duty as tracking cameras and pass through cameras, so higher quality cameras just weren't available. But why only a single color pass through camera? This really doesn't make much sense other than if you allow that the processor on the Quest Pro is so limited that even if it had both feeds it probably couldn't process them.
I think it's actually great news that there is so much optimization left to do. Seems like they should be able to improve it a lot over time, and in the meantime it's pretty adequate to enable exploration of some unique experiences.
edit: VR => MR
Until AR display tech gets significantly better I think AR based on passthrough cameras is the only sensible approach.
I also really don't know why he decided to deemphasize the perspective and depth correctness so much. He mentions it here:
> In this case, they were willing to sacrifice image quality to try to make the position of things in the real world agree with where virtual objects appear. To some degree, they have accomplished this goal. But the image quality and level of distortion, particularly of “close things,” which includes the user’s hands, is so bad that it seems like a pyrrhic victory.
I don't think this is even close to capturing how important depth and perspective correct passthrough is.
Reprojecting the passthrough image onto a 3D representation of the world mesh to reconstruct a perspective-correct view is the difference between a novelty that quickly gives people headaches and something that people can actually wear and look through for an extended period of time.
Varjo, as a counterexample, uses incredibly high-resolution cameras for their passthrough. The image quality is excellent, text is readable, contrast is good, etc. However, they make no effort to reproject their passthrough in terms of depth reconstruction. The result is a passthrough image that is very sharp, but is instantly, painfully, nauseatingly uncomfortable when walking around or looking at closeup objects alongside a distant background.
The importance of depth-correct passthrough reprojection (essentially, spacewarp using the depth info of the scene reconstruction mesh) absolutely cannot be understated and is a make or break for general adoption of any MR device. Karl is doing the industry a disservice with this article.
I also can't tell if KGuttag believes that you can simply pass the video feed through to the user and not have an absolute usability disaster. His paragraph about the "pyrrhic victory" sort of implies that not preferring depth-correct scene reprojection was an option. In my experience, that is absolutely non-negotiable in terms of shipping a device that's usable to the majority of consumers.
Another reason why folks also don't understand why passthrough is so energy intensive. You're also running an entire additional pass of steresocopic spacewarp on what is essentially four different camera feeds, on an incredibly high resolution scene reconstruction/SLAM mesh.
Would it be nice to be able to actually read text and not have distortion when objects get too close? Yes, of course. But it's not necessary.
Which unfortunately includes your hands.
But it is a big turn off to have no visual idea of what is going on around you. Just to be able to press a button and be able to see that there is someone standing around you, or you are too close to a couch, is a big win for a VR headset in my opinion. Obviously the visual fidelity will improve in the display and the pass through. But just starting to get some pass through is very nice.
Would you mind giving user-facing examples to illustrate the difference? As in something you consider AR but not MR and vice-versa.
I've googled a bit but I don't understand the distinction between "enable display and interaction with the real world but with [virtual] augmentations" and "bring virtual objects into the real world and allow you to interact with them". As someone with little AR/VR/MR experience these sound like two ways of phrasing the same thing. Thanks!
An AR example would be the classic overlay of text on a sign you are looking at with a version that is translated into your language.
[0] https://www.oculus.com/deeplink/?action=view&path=app/158709...
Where would you set Pokemon Go’s “AR view” [0] kind of implementations for instance ? (First link I found was for a headset, but I’m also actually curious how you qualify the phone only version of it)
Or what would it change if in you Gravity Sketch example each user would be seeing colors or rendering adapted to their vision/preferences making it a separate experience for each of them ?
This image show an example of it: https://www.appliedart.com/assets/images/blog/_newsFeatured/...
I hope the folks there can figure out the operational difficulties, and find a way past whatever org chart drama is the source of the dysfunction. The Quest 2 has been a delight, but enough time has passed that the limitations are starting to chafe. I would love to see a true Quest 3 made my people who actually like the things.
I can understand they try to see far to anticipate a future where they'd make a lot of money being useful, but there's such low latency and richness of perception with normal atomic real world interactions that I'm not sure we can or want to replace it with something that simply adds a paid intermediary.
It s like adding a blockchain to transact cash for instance.
[0] https://medium.com/@jas0njames/the-primacy-of-shipping-94b8e...
Moreover, "Mixed Reality" strikes me as marketing-speak. Compare the search trends for AR [1] vs MR [2]. While both terms have been in use for a long time, AR was always more prevalent, with a notable rise in interest around 2009. My first encounter with the term MR was in marketing materials for Microsoft's HoloLens circa 2015. Now compare the search trends for MR [2] with HoloLens [3]. Searches for MR don't begin to trend upward until after the introduction of HoloLens, where Microsoft's marketing materials tried to make a big deal of said distinction and the rest of the world started to run with it. As a supporting argument for the original broader usage of the term AR, I'll cite discussions of a 2007 anime series, Dennō Coil [4], which explored various examples of fantastical AR usage. Probably just a coincidence, but the search trends for Dennō Coil [4] do precede the search trends for AR [1] rather closely.
Naturally, all the usual caveats about word meanings changing over time apply, but this one is a pet peeve of mine.
[1] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
[2] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
[3] https://trends.google.com/trends/explore?date=all&geo=US&q=H...
[4] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
These early MR references use the term to mean a spectrum from "Augmented Reality" to "Augmented Virtuality". The AV term never really caught on, and the Wikipedia link for AV [1] actually just redirects to their page for MR [2]. I can't think of many notable examples of AV at the moment, though something like "Mario Kart Live: Home_Circuit" [3], as mentioned on the Wikipedia page for MR [2], might fit the bill, in the sense that reality is being used to augment the game. Uses of haptic feedback in VR might also count as AV.
This usage of MR has nothing to do with how Microsoft began using the term. It's almost as if they embraced an academic term for the sake of marketing, extended the meaning, and successfully extinguished the original academic definition.
[1] https://en.wikipedia.org/wiki/Augmented_virtuality
[2] https://en.wikipedia.org/wiki/Mixed_reality#Augmented_virtua...
[3] https://en.wikipedia.org/wiki/Mario_Kart_Live:_Home_Circuit
(Assuming by "mix depth to objects with the real world" you mean "having virtual objects alongside real objects with convincing depth alignment")
People are now using "MR" to mean something distinct from AR but GP and myself are arguing that the term AR was broad enough to encompass passthrough, overlay and any future developments.
If someone built a VR headset that used small projectors rather than flat panel displays, we wouldn't need to invent a new term for it. The AR <> VR spectrum was all we needed and MR is just muddle on top of that.
Alternatively, if you are going to just stick to one hyper general term then you can't really complain that product X is bad at Y because it doesn't do every single thing the general term Y encompasses. Language is fluid for a reason - it adapts to our needs to define and distinguish things. In this case, the Quest Pro does a subset of things quite well, while completely leaving other aspects of AR on the table. Whatever we refer to that subset as, it's not really valid to then say it's generically bad just because we don't have good words to describe what it does do.
I was a bit sloppy in that regards. I was more worried about what was going on. But when combined with the word "passthrough" to see the real world, I tend to drop back/slip to the term AR passthrough which is what this type of thing was called for years. MR, as I remember it, is a more recent term used to distinguish different types of AR. Then things flipped and XR (=AR/VR/MR) was used to mean was AR used to mean.
My guess is that late in the program, the importance of passthrough was elevated, perhaps in response to Apple rumors, but they were stuck with the hardware that was in the pipeline. The passthrough available is a big improvement over the Quest 2 for use in seeing the grouse-level surroundings, but not up to regular use in full Mixed Reality.
There are probably decades of "optimizations" left to do.
That's literally the same thing in terms of hardware. If you want to bring virtual objects in the real world, and not have weird z-sorting issues all over the place, you need to have an understanding of the real world, which in turn gives you the possibility to do augmentation.
Beside, MR is just a confusingly overloaded term these days. It can refer to:
* Microsoft's Mixed Reality brand of VR headsets (that completely lack any AR/MR features)
* filming people playing VR games in third person and mixing the game footage into it with LIV or similar software
* doing AR with VR headsets and pass-through cameras, instead of see-through optics (e.g. Lynx R1)
> But why only a single color pass through camera?
Pico4 makes the same mistake. I really don't get it. Lenovo Mirage Solo back in 2018 used two front facing cameras at a proper IPD, and while the image is still quite low resolution, having actual 3D with zero distortion made the thing feel like wearing actual glasses, not like looking at a weird camera feed. It's a ginormous quality improvement for very little extra effort. Using pass-through on that thing is still the only time I could literally forget I was wearing a headset.
I tried the Quest 2 last year mostly out of curiosity for the use cases that Meta themselves would go on to push as the primary focuses of the Quest Pro; VR productivity and virtual monitors. I was left drastically underwhelmed at the time with the resolution and viewing experience from the Quest 2 (among other things), and after hearing that the Pro wouldn’t have much of an upgrade in that department I was really curious what actual productivity focused power users would think of it.
This blog/series of posts looks to be just that, and I’ve added it to my feed reader for the rest of the series!
They suffer the same issues as other VR solutions (focusing on fine text can be hard, it shakes on your head, etc). But unlike the Quest, you can set it to “Aircast” mode which basically replaces an VR/AR experience and turns the headset into a true “head mounted monitor” - which is different than their special software. I tried it with their virtual monitor software, and it’s unusable. If you move your head, it refreshes the monitor you’re viewing at like 2fps so it’s super jerky and not smooth.
It’s not good enough to replace a real stationary monitor for me yet, but it’s probably good enough for someone constantly on the go, or for airplane usage (extra privacy). They’re also far more comfortable than a Quest since they don’t have batteries or a processor built in.
I want to add that I think they shake more than the quest because you can’t strap it in tightly like you can with a quest and a hard strap
But else, they seem great. It's cool that they draw power from the connected device, one battery less to have to worry about.
Maybe to explain why I want more than 60hz - I wouldn't buy them to be productive, but for gaming. I've been gaming with 120hz for quite a few years now, and don't really want to half my framerate. Even when gaming in VR I'm experiencing between 60 and 90fps.
I guess gaming is the only thing which prevents me from buying these glasses, for work 60hz is enough and most movies are 24fps anyways...
They had a huge media blitz recently on YouTube. They got a bunch of YouTubers to talk about how great they were paired with the Steam Deck. Apparently they extend the battery life too because the built-in screen uses more energy than the glasses.
I don’t do much gaming where specs matter (mostly co-op of casual games), and I’ve never used them for gaming. I imagine I wouldn’t mind them too much depending on the game. I bet a game like factorio would struggle since there’s a lot of UI elements that are small and it’d be hard to distinguish with the glasses instead of a big high-res monitor. Other games would probably be fine though (again except for frame rate).
Didn't even think about this use-case...Sounds fun during a plane flight, holding the Steam Deck in your lap and just resting your head the way you fell comfortable. Probably a bit jarring for the other passengers :) But also to just lay on your couch or even in bed.
> I bet a game like factorio would struggle since there’s a lot of UI elements that are small
Bet, yeah. Probably great for first-person, third-person and racing games. Games which rely heavily on UIs and menus are probably not that fun. I don't know how sharp the screens on these glasses are, but I couldn't imagine playing Factorio on a HTC Vive. Granted, the Vive is old and screens have gotten better, but it's the only reference I have.
So, this is something I think I'm pretty much missing here. Acceptable 25" 1080p monitors are pretty cheap these days; why would a potential user not just get some of those?
VR might or might not be useful for some things (I'm pretty much unconvinced so far) but the "virtual monitor" use case is just baffling to me. Actual monitors exist.
First of all, it works with phones, tablets, game consoles, computers, etc. Anything with a display output, so keep that in mind. The use-case isn’t just big desktop computers or even stationary devices.
1. Travel, you can use this on an airplane, which is nice for frequent travels, or if you want to watch a movie on a bigger screen on a plane. It also works in hospital beds, or anywhere else you may want to enjoy a screen but not bring monitors or TVs.
2. Ergonomics, you can lay down and use the glasses a lot easier than a mounted monitor overhead (and a lot less risky). This is useful both for finding ergonomic positions for general life, and also for people who may be disabled, or otherwise have some bodily issue that forces them to lay in bed (in hospital, elderly, etc).
3. It’s simply a bigger screen. It fills your vision, it’s a lot bigger than a 25” display, even if thats roughly the equivalent quality. If you want to Eg. Watch a movie or play games, it’s a better experience. I’ve heard YouTubers like it for their Steam Deck.
4. It’s a lot smaller than a monitor. If you live alone in a studio, this could save you space on having a dedicated desk and desk equipment. You can just work at your table, and it packs away to a backpack when you want to eat.
5. You could even walk and wear it as a HUD - theres a company that makes live translation software for the headset, where you can read “subtitles” of people talking.
Not sure where you heard that, the Quest Pro absolutely offers a massive upgrade in the viewing experience, specifically for the case of virtual monitors. It's mostly in the lenses but it's an absolutely huge difference.
That's not really a fair assessment of the Quest 2. The Quest 2 is just a budget (400 Euros) gaming console, and for that it works great.
A 400 dollar gaming console with a theoretical resolution of 1,832 × 1,920 per eye, is obviously not gonna perform well if you use it to emulate high resolution virtual monitors meant to sit further away from your eyes, further decreasing the perceived resolution, so you can read text and perform productivity tasks. Of course it will suck at that, but it was never meant to do that.
That's like complaining your Toyota Aygo sucks at offroading or track racing.
It's unfair because the Quest 2 is mainly a console and was never marketed as a productivity AR device. That's like trying to do your job on a PS5 and claim it's underwhelming, or take your Ferrari off-roading and claim it's underwhelming or trying ML workloads on a Gameboy and claim it's underwhelming.
You are are of course free to try those scenarios and have your opinion that the devices are underwhelming, but nobody can take your point seriously.
The primary use case for passthrough is bringing stuff into VR, specifically your hands (when used in hand tracking mode), and your keyboard. Your monitor is projected into VR by being hooked to your headset. Any other function's utility has been traded off for these purposes.
Seems obvious to me that the "Mixed reality" stuff is just marketing fluff, if you pay any attention to the technical evolution of this product line when interpreting these complaints.
For an example of this on Quest Pro, check out Wooorld. It's a fun hangout app and is arguably a much better experience in passthrough.
However, the Quest 2 and Pro apply some smart reprojection to correct this, and it feels like everything is almost exactly in the right place.
You do notice the artifacts of this reprojection when you’re holding objects close to your face, when you’re moving objects around, or when you’re looking at thin objects such as wires.
I also recall John Carmack talking about this topic in one of his unscripted talks at Meta Connect, but I’ll have to dig that up.
That's a non-issue. Lenovo Mirage Solo had a stereo cameras at a proper IPD and the immersion of pass-through on that thing is still by far the best thing I ever experienced in a VR headset. The little offset between the cameras and your eyes is unnoticeable unless you try to touch your nose or do other stuff really close to the headset. And if you really need to, you could get rid of that offset as well (45° mirror and mount the cameras vertical to have them capture the same image as your eye would).
The problem with most other VR headsets is that they either have only a single camera for pass-through or use the tracking cameras that are mounted far away from your eyes. Doing the remapping of perspective in software just doesn't give good results.
(1) There is no killer, must have game for VR. Beat Saber seems to be the most popular and mainstream game but not nearly engaging enough to convince the mass market to buy one or more VR headsets. Related, we cycled through about 5 games and none of graphics were particular realistic or immersive. I didn’t walk a way from the experience thinking wow! And note I’ve had this “wow” experience playing PS4 games on a higher end TV that supports HDR.
(2) We had a 1 hour session with the goggles and I felt quite sea sick during and after gameplay. The headset was also uncomfortable after 30 minutes or so. Playing games while sea sick is about as much fun as it sounds. It’s pretty awful.
(3) The arcade was expensive. I paid nearly $100 for 1 hour for 2 people in a MCOL area. One of the reasons for the high cost is the arcade required an employee to help us cycle through games and explain gameplay and controls. It dawned on me after the experience that my son and I would have to buy two headsets, not just one, for any couch coop play. Maybe we’d have to buy two copies of games as well? Gaming is really becoming a social experience, which is why Fortnight is such a big hit because it’s essentially free and cross platform.
VR is dead in the water unless they bring to market must-have games (or other apps and Beat Saber is nowhere close to this), solve the sea sickness problem, and somehow bring down the cost so two headsets cost about as much as a current generation console.
I think it’s also very premature for Meta to go after the business market. It first needs to convince geeks like me to buy and evangelize about VR. If anything, my experience at the VR arcade convinced me NOT to buy VR goggles. Good luck building out a new market if this is a typical reaction to the state of VR today.
b) Business market is already a large adopter of VR devices e.g. for training, warehouses, architecture/car etc demos etc. Meta is big enough to tackle both markets at once. And Quest Pro helps to validate newer technologies that will trickle down into their cheaper models.
c) Motion sickness is better on newer devices and from research videos Meta already understands what are the causes of it. It's now largely an engineering question of how to get solutions translated into shipping products.
I would, if it was high quality enough to be comfortable for long term use, and there was decent enough virtual desktop software to have many floating windows of all the software you use.
I strongly disagree here. The enterprise market is relatively cost-insensitive, and so it's a great vertical to target. The partnership with Microsoft is strategically very sound; if you can get Teams inside the Quest headset, then you can plausibly replace every lightweight Windows laptop with a Quest Pro (i.e. for road warriors doing spreadsheet/doc/email/collaboration work, not developers).
The question is, how far away are they from replacing monitors with VR displays? I suspect this generation is not the one that wins that battle (though Immersed was making this work on Quest 2, so maybe), but it plausibly could be next generation, in which case this is the correct time to be building the MS partnership.
On gaming -- we are multiple generations away from VR being more than tech demos, IMO. Beat Saber was fun, but pretty much everything else feels boring after 30 mins. I agree with your assessment that VR gaming is quite underwhelming, but I disagree that they need to win over geeks to crack the enterprise market; they just need to get Nadella on stage to provide a compelling use-case to enterprise users, talking in the enterprise language.
The unlimited monitor argument is only relevant if the resolution, battery, ergonomics, pass-through, etc vastly improve. So at this stage it's still a fantasy.
> Potentially much better collaboration, since you can be “in the room” with your team, see their facial expressions, etc.
Seems like you are talking about some theoretical future AR or MR headset. As of today MQP can not do any of that.
The Quest2 is a massive pain to use for an extended period of time, but I think it totally proves it’s possible. You can browse, use teams, anything browser related easily. I haven’t used the QPro but I imagine it’s a nicer experience (except for battery).
Resolution is a concern, and IMO they should have made a high res but otherwise low spec version for the Pro. Maybe they just wanted to flex all the possible tech in research (eg the fancy controllers). The road-warrior angle is totally a real demographic and I think this could be a real win in a few generations (I say 3-5 years at current pace). I suspect it’ll first look for like the NReal Air based on using it, since it supplements an existing computer with a bigger screen. I think another demographic is the opportunity to kill the WFH desk. Any table becomes good enough, or a treadmill… or anything. When Covid started, I had to go buy a desk and monitor since I didn’t use one at home. I know lots of coworkers who lived in small apartments and had to make big compromises to wfh in their space.
My gf is the sweetest, non-confrontational human I know. She once cried because she saw a dead frog on the street.
After 5 minutes in Blade & Sorcery she was brutally decapitating already dead enemies while giggling, throwing them off cliffs while watching how they fall down etc.
When she took off the glasses after 10m we both just had to laugh, as she was heavily breathing and sweating from all the fighting. One of the craziest things I've ever seen her do.
Same things has happened for the two other people I put into this game, and for myself.
Just wanted to chime in because I agree with you about HL: Alyx. Currently playing it, and it's the best thing I've ever witnessed in VR. Crouching behind a virtual wall because enemies are shooting and throwing grenades at you never felt so terrifyingly good.
I've never felt such a consistently visceral adrenaline response from a game before.
I can't really speak to what Meta's doing, but as somebody who pre-ordered a Vive at launch and has had some of my most compelling gaming experiences ever in VR, I think you should recognize that you aren't really in the target market for higher-end VR gaming. The absolute size of that market is certainly a concern for the continued development of VR tech, but so far it seems to be doing alright as a relatively niche gaming sector (see: the Valve Index).
> somehow bring down the cost so two headsets cost about as much as a current generation console
You can get an Oculus Quest 2 for $400 + tax.
It's so good that it ruined VR for me entirely. I can't play any other VR games, they all just feel like tech demos or shoddy old Unity asset flips.
It's scary good and immersive. Try it, it will blow you away.
I've installed the HL2 VR mod and played that a few times, but without teleporting movement I got sick quiet easily. I saw another commenter saying you have to acclimate yourself to that, so I suppose I will try that in between sessions of Alyx.
Here's to hoping someone makes something as polished and as beautiful as HL:A
It's useful to hear the experience of casual testers, enthusiasts will praise the technical achievements, but that doesn't mean it's a good product.
Part of the problem here is that you tried to push through it. Basically the only way for someone who immediately experiences VR sickness to acclimate is to stop after 5 to 15 minutes, give it a while to go away, return to it, and repeat the cycle a couple of times until it stops completely. If you just try to push through it, that actually makes it worse by aggravating the reaction you're having.
Of course, an arcade making the (poor) decision to brush over any issues isn't going to tell you this.
> The headset was also uncomfortable after 30 minutes or so.
I'd bet they were using first-party headstraps, which for whatever Godawful reason are almost universally uncomfortable across every brand. Companies like BoboVR produce a different style of much-more-comfortable 'halo' strap arrangement that rests the weight on your head more like a hard hat instead of clamping it to your face.
> It dawned on me after the experience that my son and I would have to buy two headsets, not just one, for any couch coop play.
There are a couple of games that mix one VR player with one or more non-VR players. Carly and the Reaperman, for example, has one player in VR manipulating the environment, and the other playing a traditional platformer. Another example (and an excellent party game) is Keep Talking and Nobody Explodes, where one player is defusing a bomb in VR but only the players outside VR have the bomb defusing manual.
It’s also one of those types of games that’s completely different in VR than regular desktop. Without VR, it’s a fairly fun game with a really good learning curve to it. On VR, it’s a completely different game; the atmosphere is amped up by like 1,000 and it becomes the scary game it was meant to be. I wasn’t even able to complete the tutorial on VR.
Considering the context of adoption obstacles, this is worse than VR just making people feel nauseous.
If VR made you feel bad, be sure to both remember how bad it made you feel, and also voluntarily make yourself feel bad again, repeatedly, until (maybe) you won't feel as bad anymore, and then you can have the day-1 experience? Am I tracking this correctly?
I recently installed Skyrim SE and played it a bit on my new 4k HDR OLED. Then I wandered off and installed the Skyrim VR on my gaming PC.
As you say - The 4k HDR remaster is "WOW!". The VR version is more, "Neat."
And this is a game where people - rightfully - talk about wanting to live in the game.
I don't know which headset they used but I do know we each had an entire PC in a backpack strapped to us.
The experience of a game like Alyx on the Index/Quest is about 100x better.
Also, I hear people say that video games are dumb and pointless sometimes, which reminds me that I might myself not realize when something has incredible potential. I think a lot of people might never see the potential in VR, when it’s already there with millions of people using the Quest already
I think the best VR game of this generation is Beat Saber.
And I 100% agree with you about VR being here. Meta has sold something like 10 million headsets. VR is here and it’s doing well. It’s probably not going to overtake other forms of gaming and that’s okay. It doesn’t have to. Ten million users means it’s a big success.
This reduces significantly wit time. Also on the first time you don't know which setting works for you, having correct IPD, straps etc. When I used my quest 2 for the first time, I could hardly use it for 10 minutes, and now I could use it for 1 hours. I think if I use it everyday even that limit will be increased.
Would you personally consider using it to "work in a virtual office" for several multi-hour stretches every day?
This is the main issue with it right now, and part of the problem has been that to get AAA quality games on it you need to teather because none of the existing headsets have anything close to the power needed to run a decent game, not to mention they get very uncomfortable, very quickly.
I mentioned elsewhere about Apples apparent headset thats possibly coming out this year. This _could_ be where Apple can shine. Whilst they've not exactly got a great rep as a gaming power house, if their headset does indeed end up using their M2 architecture it means its going to be extremely power efficient and very light weight - solving two of the biggest issues. But it also means its got graphics capabilities for most modern AAA titles.
Obviously they'd need to work with a few studios to incentivise them to develop for their OS, but given their history of entering and significantly disrupting markets I'm sure they'd have no problem bringing along a few decent studios and getting them to work with their headset.
Combine that with their existing App Store and Arcade+ services and you've got thousdands of developers itching to get their games released on the platform.
If they can successfully replicate the popularity of their iOS developer ecosystem it would completely destroy Meta and their quite frankly crap little selection of apps.
---
For what it's worth I found initially I felt pretty awful using it, it does take time to adjust and you're best using it in short bursts for the first few weeks.
Recently I decided to do something stupid and try out the F1 2022 game on it (via cable plugged into pc), expecting it to make me feel extremely sick. I was pleasently surprised at how amazingly good it felt. I was able to complete a couple of races without feeling sick at all - its definitely an acclimatization process that the manufacturers need to make a lot clearer is needed.
VR device is for using at home, slowly familiarizing yourself with it and using it for short bursts until you got used to it and don't feel any bad effects.
I did this and its amazing for games and especially simulators like DCS.
The combination of such realistic feeling gameplay and multiplayer is way more compelling than I thought it would be.
Slight nitpick:
Those would be the HTC Vive Pro headsets. The "HTC Vive" came out in 2016 and is a pretty shitty device by today's standards. Also, it retailed for $600.
> We had a 1 hour session with the goggles and I felt quite sea sick during and after gameplay.
This varies from person to person, but most people only get sick when playing games where the real-world movement doesn't match game-world movement. For example, most people can play Beat Saber with no motion sickness problems, but will feel sick almost instantly if they play a racing game or a game that lets you walk by using a controller input.
I've had VR since late 2016 and this still rings true. I can play PokerStars VR for 4+ hours and my only complaint will be the weight of the headset starting to feel overbearing. On the other hand, 2 minutes of Assetto Corsa and I feel like I'm gonna lose my lunch.
> The arcade was expensive. I paid nearly $100 for 1 hour for 2 people in a MCOL area. One of the reasons for the high cost is the arcade required an employee to help us cycle through games and explain gameplay and controls.
Shitty experiences in VR arcades are exacerbated by the fact that they're usually staffed by people that just needed a job, and not VR enthusiasts that care about sharing their excitement for the platform.
(EDIT: Also, as someone else mentioned, having IPD configured properly is crucial. If the staff aren't setting IPD, then it can create headaches and sickness.)
> solve the sea sickness problem
It's a difficult problem to solve because the core issue is the mismatch between movement in the worlds as mentioned above. When your eyes detect movement, but the vestibular system in your inner ear says your not, your monkey brain thinks you ate something poisonous and creates the urge to purge.
A company called Otolith Labs has supposedly created a device that potentially solves the problem (https://otolithlabs.com/vertigo/) by making your vestibular system essentially send static, but they're targeting it to sufferers of vertigo rather than VR players.
> and somehow bring down the cost so two headsets cost about as much as a current generation console.
As others have said, the Meta Quest 2 is $400. And it's PC VR compatible, so if you have a gaming PC, you can use it with that.
> I think it’s also very premature for Meta to go after the business market.
We're still years from business use for VR aside from maybe virtual meetings with avatars. I think the biggest limiter is resolution. Text needs to be decently large to be readable in VR. I have a 1440p monitor that only uses what, 45 degrees of my eyes' FOV? When I think of what I would use VR for, I'm thinking essentially unlimited monitors. I could have several pages of documentation visible at any time.
But someone in our office had a Quest Pro and I was blown away. It's the first time I've ever felt like I wanted to use or build thing in or for VR, and it was the Mixed Reality mode that make it like that.
Unlike this review, I found mixed review very impressive. I suspect it was because I was viewing it primarily as a VR device with just some limited capability for real world interaction as opposed to an AR device. I think when viewed through a VR-first lens it's very impressive.
I still don't know what I'd actually want to use it for, but it was the first time I felt like Meta might be actually onto something.
From these results, it looks like you’ll still be SOL if you forgot which key on your keyboard has the backtick, or try to quickly glance at your phone, or check your room’s air quality or temperature to adjust as needed.
From the pictures it still looks twice nicer than the Quest2, but we’re so far away from something actually workable.
Also VR Chat (also an app) has been awesome for connecting and talking with new people. Environments from orbiting high above the earth in space to a rooftop restauraunt in Paris make it fun to spend a few hours with new people.
As other have said, inside other VR application there is a lot of "neat" "potential" but hampered execution due to software, hardware, or both. I always tell people we've made some gigantic leaps but we're still in the early stages.
1. 30 Hz frame rate and 50ms photon-to-photon latency sound so bad that I'm surprised they launched this at all. I would be very interested in trying this out to see how (un)comfortable the experience is. What I would have expected is pass-through feed rate synchronized to display frame rate (i.e., 90 Hz) and photon-to-photon latency no more than perhaps 2 frames (22ms). At 90 Hz and 11ms you can actually play ping-pong while wearing a pass-through headset.
2. I can relate with the exposure, saturation, and noise issues. The existing well-tuned video ISP pipes aren't suitable for low latency, and building a new one that achieves low latency and gives a good picture in indoor lighting from an essentially smartphone sensor with exposure time < 11ms and doesn't add multi-frame latency is HARD.
3. Above points to one possible explanation for the 30Hz capture; maybe they used a cheap sensor that can't produce usable picture in < 11ms exposure time (in dimly lit indoor environment) and thus had to go with 30Hz capture. Would be interesting to repeat the experiment in well-lit (outdoor) environment to see if the capture pipe kicks into a higher framerate.
4. I wonder if they do any warping to correct for the camera positions and how well that works. For AR-focused headset the ideal pass-through camera positions would approximate eye positions to minimize need for distortion. Even shifting capture positions a few centimeters forward from the eyes (which you'd need to do if you don't replace eyes with cameras or play tricks with optics) creates a subtle but noticeable effect unless corrected.
The main thing is knowing maps and the radio. A team that uses the radio well and can call out locations will destroy a team of sharp shooting lone wolfs.
There are people who take it serious and those that are trying to figure things out. It’s all good. People will teach you things. Also the competitive lobbies are marked as such, so just avoid those.
The tear and color lag on this picture of a moving chart is damning: https://i0.wp.com/kguttag.com/wp-content/uploads/2023/01/MQP...
As is this quote:
> I also ran a small experiment where I sent a camera flash off in a dimly lit room and measured the delay with light sensors using one sensor to trigger an oscilloscope’s capture and the second light sensor to capture the display response. It typically took a little over four 90Hz frame times (~40 to 50ms) for the flash to show up on the image.People who capture Quest footage using scrcpy are well aware of this downside compared to using the built in recording feature which doesn't have the same tearing issue.
Here are some things on the horizon for us to finish before we release test units to YouTubers:
- Add an IPD adjustment mechanism[2]. Currently, our prototype's IPD is locked in place until we add this functionality, which makes it really poor for testing since there's lots of variation in IPD.
- Incorporate our front-facing cameras for our AR Mode. On this front, we recently shipped our Camera Boards to PCB fab[3]. Note that our cameras support 2448x2048 per eye resolution, and are placed more directly over the eyes than the Quest Pro (avoiding the issues discussed in the parent post). For a fuller comparison against the Quest Pro see[4].
- Replace our FPGA eval board with our own custom FPGA. (Just started on this; will likely discuss it during our next blog update).
- Integrate our detachable compute pack to the back of the headset (right now our compute pack is just dangling).
So all in all, while much of the core functionality is there, our headset is still too unpolished to send to YouTubers.
Also: I understand your skepticism RE preordering. We try to keep a consistent stream of blog updates to at least let people know what's going on engineering wise.
[1] https://simulavr.com/blog/first-glimpse-of-review-units/
[2] https://simulavr.com/blog/simula-one-auto-ipd-adjustment/
But I love my Meta Quest Pro because I love the Oculus 2 for fitness VR and the Pro is incrementally noticeably better
And the humble AR is fantastic for minimal interactions like physically orienting or real world communication in a pinch (ironically!)
it vaguely resembles being there ... because I am?
I have yet to run a Meta social VR or AR app. Too scared I will run into a creepy Zuckertar!!!
For me, the only time I've seen it pop up is when I step out of my "zone". Rather than just flashing red, I get a vague understanding of where I'm standing and who's nearby. I've never had a problem with it at all, even though I acknowledge the picture isn't particularly great.
The point of the article is that this experience, while good, is juxtaposed with Meta's marketing claim that it's a mixed reality device. You're saying the device is good at the thing the author says it was probably built for. Mixed reality implies that you're seamlessly switching between environments, as if there wasn't a distinction.
It's bad at doing a thing that's window dressing.
The article implies they’ve been hyping up AR (which matches my tiny knowledge of the product) and that seems useless if the Reality part of AR is blurry.
The article is effectively lying. Try searching their page [0] for augmented reality. It's not there. They refer to "mixed reality" specifically because that is actually different to AR, specifically in the sense that "reality" is not the primary focus of the experience.
> With breakthrough high resolution mixed reality you can engage effortlessly with the virtual world while maintaining presence in your physical space in hi-def color.
People seem to just not understand that mixed reality is different to augmented reality?
I can't help but feel that this is a distinction without merit. It appears to be jargon designed to muddy the waters and deflect criticism.
It's going to be interesting to see what happens if the rumors about Apple's being extremely lightweight are true - it wouldn't be surprising if it is given their tiny but efficient SOC design. This is what will kill Meta if they arent careful. They're about to get a competitor who's able to deliver performance levels significantly higher, and at lower power consumption and weight.
- being able to see in extremely wide spectrum bands compared to human eyesight
- being able to see in higher dynamic ranges than the human eye
- being able to magnify little details and thus see with a higher spatial resolution than the human eye
- all of this while enhancing your field of view with lots of other interesting information
edit: minus IR/thermal
(Yes thermal cameras exist, but not really as an AR overlay)
"The IVAS adds night-vision and thermal imaging sensors to the usual HoloLens heads-up display, and is ruggedised for military use."
https://www.army-technology.com/news/us-army-to-work-with-mi...
IMO the more interesting business cases would be around simulated environments even as simple as viewing 360 videos for places where workers would be expected to know what's going on when they enter.
Thanks for the detailed analysis!
For example the the pix comparing the headset to high-end Canon VR180 .. that suggests maybe the viz should just be bigger?
Great write-up about the faults, but there’s gotta be a work task that can work? Beyond beat saber?
If we take 60 pixels per degree as the Apple "Retina" standard, and if we assume that the field of view of the Apple VR device is ideal (120 degrees), then we would need a horizontal resolution of ~7k to match my setup at the "Retina" standard. If you compromise a bit on FOV, you can squeeze it down to 6k, which is within the reach of "today's" current display technology [2].
The Apple VR device is rumored to be powered by some custom silicon with foveated rendering optimizations (render only what you are looking at with the highest resolution) and is maxing out an M2 class GPU with unknown # of GPU cores on the device per The Information article. A doubling of GPU performance may get us to the 6K display per eye, optimistically the v2 of the Apple VR device?
[1] https://www.theinformation.com/articles/inside-the-tech-powe... [2] https://www.roadtovr.com/pimax-reality-12k-qled-announcement...
A few generations and it’ll be doable.
- it's helpful to have it be gigantic - bigger than a real monitor could ever be
- you have a real need for more monitors than would be practical to buy or position in your work environment
- you will be highly mobile and moving around so much that it's going to be better to take a headset with you than try to get monitors everwhere
- you really would like to collaborate with someone else who also has a Quest Pro, in which case you can work together without being physically together
- you highly benefit from / enjoy working in alternative environments (eg: being highly focused, free of distractions, relaxed, surrounded by beatiful scenery etc)
I actually find the latter point is the biggest reason I do it. Even when I'm surrounded by distractions I can go into a completely calm virtual environment and get a crap load of work done.
You can get something up and running - but it feels akin to staring at a 1080p screen with a weird delay. I felt way more productive just looking at my tiny laptop screen.
Your mileage may vary, obviously. I really liked the idea of working in a distraction free virtual environment, but I feel like the tech isn’t there yet unless you’re an enthusiast whose willing to put up with all of the strangeness.
So no, for most common uses this is nowhere near good enough. It's like having an old 720p LCD TV on your desk. It works for casual web browsing and movies, but far away from the image quality of a modern 4k monitor.
Nreal Air at around 49 pixel-per-degree is the closest thing to a monitor-replacement you can currently find, even has a plain HDMI input. Does however come with a lot of other short comings (no 6DOF tracking, monitor image not tracked, low FOV).
Cameras like the Sigma FP are built around the heat sink and have vents all around to dissipate the heat during long recording sessions. Most typical DSLRs simply have recording limits e.g. Canon R6 = 30 mins.
Pretty big issue for a device they are aiming to be around your head for hours at a time.
https://www.fujirumors.com/yes-eu-import-duty-reason-fujifil...
It’s almost like every Apple product has been developed and refined for the future AR/VR moment. Custom silicon. Fast camera HDR. LiDAR. People/scene/room/boundary detection. AirPod Max fit and finish. Spatial audio. 50 other things…
All in the service of AR/VR.
Apple is absolutely gearing up to dominate AR by having a sleek set of glasses that use your iPhone for the graphics rendering. And they'll succeed.
[0] https://www.macrumors.com/2023/01/03/report-reveals-wave-of-...
I cannot fathom apple releasing a waist mounted battery pack like the article suggested.
Absolutely refuse to accept they’d do this.
AR is very hard to do good enough. HoloLens, MagicLeap and others are pathetic.
I hope I am wrong.