Comment by acdha | Hacker News Reader

Comment by acdha | Hacker News Reader

Parent Full thread

Are you asking for citations to research papers? It’s briefly explained in context but basically your eyes capture a ton of data and process it quickly (different types of data are processed at different speeds: your brain can react to motion before you recognize color or identify an object). That has millions of years of evolution behind it, and our entire lives are spent training expectations on how that works and especially how it interacts with other senses or what we do.

That leads to several hard problems: you don’t just need to generate a high-quality scene, you need to render it twice because your brain is used to the distance between your eyes (16K==8K per eye) and because your position isn’t static it needs to update very quickly when you move because your brain is very used to how that works in real life. You have a budget measured in single digit milliseconds to that or a fair number of people will get motion sickness and many more will notice the lag and find it detracts from their experience.

Here’s a good overview:

https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

> (16K==8K per eye)

That's not how Ks work.

And your explanation doesn't cover why the minimum number of cameras would be 12.

It’s also not how eyes work because your visual resolution isn’t consistent across the field of view, saccades are a thing, etc. but it’s serviceable enough to get the point that you’re not just rendering a high-resolution scene but two of them requiring different processing for a fair chunk of the pipeline (consider how much geometry is only visible to one of them when you walk around a corner).

> And your explanation doesn't cover why the minimum number of cameras would be 12.

This is covered in the article. The key thing to understand is that there are multiple tasks using cameras: AR needs to track the world around you, you need eye tracking to handle saccades and conscious direction, and tracking hands, arms, legs, facial expressions, and body position accurately requires multiple cameras with enough parallax to build 3-D models which are accurate enough to generate realistic, stable imagery. People will really notice if things bounce around or deform or if you punt on some of the proven and end up with Meta’s leg problem.

The key point to take away here is that AR/VR aren’t one hard problem but a collection of hard problems which all need to be solved for most people to find the experience compelling — all with tight physical constraints.

> This is covered in the article.

Nothing explains why 12 specifically. And if 12 is a minimum, what's the optimum?

> The key point to take away here is that AR/VR aren’t one hard problem but a collection of hard problems which all need to be solved for most people to find the experience compelling — all with tight physical constraints.

And it's a pretty bad point because it only works if you constrain yourself to head-mounted (including battery) standalone devices. The article gives a nod to other ways of doing it, but otherwise ignores them for all its main arguments.

And you don't need optimal to be compelling.

> And it's a pretty bad point because it only works if you constrain yourself to head-mounted (including battery) standalone devices. The article gives a nod to other ways of doing it, but otherwise ignores them for all its main arguments.

The article isn’t a complete survey of the entire field, it’s talking specifically about the most popular approaches and extrapolating what the author believes a breakthrough popularity device will look like. It does mention other options such as Apple’s rumored support devices or how industrial applications can take advantage of their environment. The challenge here is price: these systems are expensive and getting people to buy them means they either need to be cheap (impossible) or give a lot of value, and companies are assuming that means something you can easily move around in and use for long periods of time (not many people are going to set up a VR room, buy a headset and sensor array, and a pricey PC).

While it’s true that it doesn’t need to be perfect, some of the hard problems spoil the illusion - people get sick or suspension of disbelief fails - and it has to be enough better than the widespread mainstream entertainment systems to justify the much greater cost. Currently we haven’t hit the point where many people, even many self-described fans, use these systems heavily and that makes it an expensive hobby.

> Currently we haven’t hit the point where many people, even many self-described fans, use these systems heavily

How many minutes per day would you consider “heavy usage” given the current entertainment/gaming focus?

It’ll vary depending on the person (e.g. a single 20-something has a lot more time than a 30-something parent) so I’d put it in relative terms: replacing console/PC gaming or Netflix. I was in particular thinking of how many people I’ve heard comment that they thought they’d use their VR headset more but it was just enough extra hassle that it stayed on the shelf.

This ties back to the price question: this currently costs noticeably more than console or even PC gaming, both of which are more social. A mainstream product needs to offer something you don’t get there to justify the extra money.

Replacing another entertainment source is definitely a clear sign of value, no argument there.

But I think you’ll find that your priors are quite a bit off at least in terms of “heavy usage” of VR though. Without any snark, I highly suggest talking to people who work in the industry if you’re curious.

Note that a new Quest 2 costs $400 with the price hike (previously $300, and sometimes as low as $250 when on sale). Compare that to $500 for a PS5 or Xbox Series X. I don’t think it’s correct to say that VR “costs noticeably more” or doesn’t have something extra to offer; don’t discount the novelty factor of VR to move sales.

Look, I’m not saying there’s no value or that it hasn’t had some big advances but simply that it’s uncommon for something to be really popular and nobody talks about it. I’ve heard at least an order of magnitude more “I thought I’d use it more” comments even from early adopters, which suggests to me that it’s still waiting for the Wii Sports / Pokémon Go moment where something clicks.

The Quest 2 is reasonably priced, yes, but every review mentioned drawbacks (often due to Facebook rather than the concept) so the question of satisfaction per dollar is still pertinent. The people I know who are VR fans all use PCs which is where my cost priors come from - high-end gaming system rather than console level - but hopefully that’s shifting. I was definitely expecting this to be more noticeably popular by 2023.

Just for reference, Quest pro has 5 cameras for position tracking, 3 for face, and 6 for the two controllers (3 each). Complete full body tracking would probably require some more in the controllers.

Assuming inside out tracking, redundancy is required, since you can make the majority of the cameras useless by standing in the corner of a room with white walls. Features are required for tracking.

Maybe the future is LiDAR, or something new, or old (military gets along with IMUs).

Ah, so we're talking specifically about inside out tracking without lighthouses. In that case the camera duplication makes that number make sense, but it could have used more elaboration.

> without lighthouses

Well, we're talking about the future. The Quest 2 implementation proved that the future is definitely not lighthouses [1], especially for mobile, which is the neccesary future of AR. Lighthouses are a technology from a time when there were no SOCs with the hardware required to support the cameras. In the end, we'll probably end up with a mix of SLAM cameras, and lidar.

Full body tracking is all that's keeping lighthouses relevant. Maybe that goes away with more cameras. :) If you have enough cameras/lidar, you get tracking of all world objects, not just those with sensors attached.

1. https://dl.acm.org/doi/fullHtml/10.1145/3463914.3463921

> The results reveal a significantly higher accuracy for the Oculus Quest 2 compared to SteamVR Tracking in the height of a tracked object. Furthermore, the Oculus Quest 2 tracks its position with substantially higher precision than SteamVR Tracking.

> Full body tracking is all that's keeping lighthouses relevant. Maybe that goes away with more cameras.

Only if they're cheap enough, but if you're in the situation of "cameras are so cheap we'll put in 40" then the idea of "minimum 12 cameras" is extremely not an issue.

Reply on news.ycombinator.com