Hacker News Reader

> no developer is paying attention to proprioception

That is not true at all. Proprioception is top of mind for every serious VR developer. Outside of poor software performance, it's the #1 cause of simulator sickness. Short of hacking the vestibular system, there's nothing we can do about the lack of virtual acceleration cues. So we design within the constraints we have. It's why you see teleport locomotion. It's why developers get into such arguments over "smooth movement" with gamers who think they know better. Lack of proprioception with virtual movement is a huge problem that we would love to solve, but laypeople have an irrational fear of a little extra electricity getting piped through their ears, so I doubt we're going to see much change there.

Fixed-view, wearable displays violate proprioception even more than VR HMDs, because they don't even give you the real acceleration cues you get from actually moving your head.

As for your question about distance, I suspect you're looking for "accommodation". Accommodation is the physical distortion of the eye to change the focal distance of the lens. Combined with vergence (the distance at which our binocular vision rays converge), the two systems work together to give you a sense of how far away something is (among other things, like atmospheric hazing, parallax, and trained memory of similar objects). Most modern VR systems lack the ability to account for accommodation, leading to what is called "vergence-accommodation conflict". It can be a source of eye-strain for some people, but it tends to be adaptable over time.

Note that I said "most" modern VR systems. It's an active area of hardware research, as the solutions--light field displays and varifocal lenses--are large, complex, and therefore expensive. The latest developments in "pancake" optics, where focal distance for the lens is folded in on itself via a clever system of internal reflectance, has a possibility of bringing the size down, but unfortunately not the complexity or cost.

So again, it's really not fair to say developers are "ignoring" these problems. We are all extremely aware of the them. There are just some things you can't do anything about without a major breakthrough.

> there's nothing we can do about the lack of virtual acceleration cues.

I think we're not talking about the same thing. I don't see how it could be a cause of simulator sickness, which I thought had to do with eye strain from focal points not being correct, and the inner ear.

Proprioception is a sense like the ordinary 5 senses, the perception or awareness of the position and movement of the body. It is what makes our body feel like our body. Dissociation, caused by dissociatives, can cause inanimate external objects to feel like they are part of us, which could be considered chemically hacking proprioception. The wooden arm illusion is another way to hack proprioception using touch to reinforce the sense of it. But VR should be able to do this without chemistry or touch, because visual clues reinforce the sense of proprioception.

VR can really be the wizbang killer app, but it isn't. Maybe they show you a virtual arm and you see yourself manipulate it, but it doesn't feel like your arm (I am not talking about the feeling of touch). That is how they are ignoring proprioception. And it doesn't have to be an arm, because in VR, you don't have to be human. It should be possible to blow the wooden arm illusion out of the water with VR, and have ten arms, and they all feel like they are distinctly your arms. But we don't see that, and no one is working on it. All we get is a stiff arm without natural reticulation.

So it is fair to say developers are ignoring proprioception, because they are unaware of their own sense of proprioception (like knowing where your hand is without looking at it). Proprioception should be of primary concern in Virtual Reality. They should toss everything, and start all over again with inducing proprioception as the prime driver of the technology.

> As for your question about distance, I suspect you're looking for "accommodation". Accommodation is the physical distortion of the eye to change the focal distance of the lens. Combined with vergence (the distance at which our binocular vision rays converge), the two systems work together to give you a sense of how far away something is (among other things, like atmospheric hazing, parallax, and trained memory of similar objects). Most modern VR systems lack the ability to account for accommodation, leading to what is called "vergence-accommodation conflict". It can be a source of eye-strain for some people, but it tends to be adaptable over time.

Interesting, but complicated. Should be called "apparent display distance," or something simple and easy to understand. Still, gives me more information. Appreciated.

A non-trivial number of VR users experience phantom sense of varying strength, including myself. Proprioception is not the only thing being consulted here, seeing someone touch what visually seems like your face is a common low-level manifestation of this. It's possible to be in an avatar and have someone touch the avatar's tail and somehow weirdly you can feel like "your tail" is being touched (albeit you feel it in your tail bone).

Maybe the technology to better integrate more brain variants will eventually exist, but current VR tech already has a niche to cater to and develop from.

> A non-trivial number of VR users experience phantom sense of varying strength, including myself.

You have elegantly spotlighted the problem. Call me when they all experience it, always and every excursion into the virtual. Then we'll know VR developers have finally started paying attention to proprioception.

Even this, what you are saying about "virtual arms", has been tried. Inverse-kinematic self-avatars were a big thing within the industry about 5 years ago. And it was... largely not that important. Without a tracking point for the elbow, it's nearly impossible to do IK for limbs. Remember, a double-pendulum is a chaotic system. A small perturbance can lead to dramatic divergence between simulated and actual state in chaotic systems.

That said, it's still possible to get body sense in VR, for the head, main torso, arms, and especially hands. It's hard to do. And when things are hard like this, there are economics of scale to consider. For example, the Quest 2 has native hand tracking built in, but not a lot of other VR systems do, and the ones that do often have dramatically different APIs (though OpenXR helps a lot here, which shifts the problem to whether or not the system in question has an OpenXR-compliant driver). Do you go all in on one platform and support its input modalities to the maximum degree? Or do you try to target the common denominator between a broader swath of the market.

Non-VR consumer software developers get to develop their software in an environment of operating systems that have well-established UI and UX metaphors. You make a button in WxWidgets or UWP or SwiftUI or HTML and that button comes with a LOT of functionality that you don't have to program yourself. We don't have that in VR. To make a button in a game engine for a VR game you basically have to implement that entire thing yourself. Oh, the game engine probably gives you a raycaster to be able to find objects, but you have to program up where that raycaster should be firing, which objects you should be interested in, and what interactions hitting the buttons on your controller map to how the button in the view reacts. To a large degree, every VR developer has to reinvent the world when they make something. We're working on the very edge of performance that these systems can eke out.

We don't have shared UI metaphors, but not for lack of trying. There are several projects underway attempting to do just that. MRTK and StereoKit in particular have put a lot of work into just that. But they aren't for everyone. They can't do everything. It'd be like expecting Call of Duty to be implemented strictly in Win32 controls. And the input modalities are so much more diverse and non-standardized than desktop mouse-and-keyboard that I doubt there will ever be a one-size-fits-all UI toolkit for VR.

I reassert that you are not being fair to the VR industry. Everything you have talked about so far has been tried and either found to not be worth the complexity or not possible to do within the current constraints.

> it's still possible to get body sense in VR,

Accidentally stumbling upon proprioception is not what we're after. We are after being fully proprioceptively immersed within seconds of jacking in, and until developers place proprioception front and center in their development of VR, accidentally stumbling upon it is all we can expect.

I don't have enough prior knowledge that I can't tell your definition of proprioception is correct or not, but I know that forward kinematics is completely neglected and that should at least be included in your version of it. Even teleop robots weirdly don't have full limb/joint angle capture nor cares about forward/inverse kinematics agreement, just inverse with control handles.

> Fixed-view, wearable displays violate proprioception even more than VR HMDs, because they don't even give you the real acceleration cues you get from actually moving your head.

This is unfortunate, I have severe motion sickness; Can't even play Minecraft or any FPS. So VR headsets are out of question.

I was waiting for display glasses like Lenovo's T1 for years now[1] due to its accessibility benefits. From what you say, It seems like I would be sick if I use display glasses too until the proprioception issues are resolved and I'm glad that it's being seriously researched.

[1] https://needgap.com/problems/16-wearable-low-latency-display... (Disclosure: My problem validation forum).

Are you saying that you have tried VR headsets and find that they are bad for your motion sickness, or that you presume that they would be bad because of FPS games?

If it's the latter, there is a small chance that you would not have the same motion sickness problems with a VR HMD as you do with FPS gaming. Most of the headsets on the market today use 6-degree-of-freedom tracking for the head and hands. So if you move your head, the view does update accordingly. Actually, I'm not even aware of any headsets that are still available for purchase that have 3-DOF head tracking, though there are a few like the Vive Flow that have only 3-DOF controllers.

Headsets like the Quest 2 and Valve Index also have extremely high screen refresh rates (120hz and 144hz, respectively) that have practically eliminated for most people most of the motion sickness problems that can be attributed to the hardware alone. Granted, there are still some dodgy apps out there. I would avoid anything based on 360 video content in your case. Hell, I don't really have any motion sickness problems and I avoid 360 video content, just because I find non-interactive stuff to be a poor experience overall.

I wouldn't go out and buy one right away, but if you run into someone with a Quest 2, I'd encourage you to try it out. Supposedly, Meta is about to release their next iteration of headsets, whatever "Cambria" or "Quest 2 Pro" or "Quest 3" are supposed to be, which should be even more powerful.

The point I was trying to make in my post was that GP didn't know what he was talking about and that devices like the Lenovo T1 would be a lot worse for the kinds of issues he was mentioning than the VR headsets he was complaining about.

I just presumed that VR would suck for me since I couldn't play any FPS and didn't try any real VR headset.

Thank you for the insights, It makes perfect sense for why VR headsets with greater refresh rates and 6-DOF might not make me sick. I'm going to try it when I get a chance.

> It is an obvious idea. AR/VR is hard. Dumb displays are not. I think we should have had similar products a decade ago, and if I'm not mistaken, these exist in cheaper SD versions from noname manufacturers.

These products have existed for decades. I can't find the one I remember seeing ads for during the 2000s, but Sony had one in 2011 called the HMZ-T1, they had a much earlier model called the Visortron. I think the widely advertised ones were maybe iTV goggles, but I can't find any history on them. Typically advertised as equivelent to a large screen tv farther away.

If you want a demo of a real product, call around to children's dental offices. Many of them have (or have had) these to help distract kids during exams and procedures, although mounting a tv above the exam chair is also popular (a lot easier with a lightweight lcd or oled than a crt or plasma)

Wikipedia (https://en.wikipedia.org/wiki/Glasstron) says the lineage is Visortron (????), Glasstron (1996), HMZ-T1 and HMZ-T2 (2011), HMZ-T3 (????, maybe also 2911)

https://www.digitiser2000.com/main-page/the-first-playstatio... claims the Visortron was never released commercially.

I had a Sony Glasstron in 1999 that was basically the same idea but TV-resolution instead of 1080p. It was fine for playing games and watching movies. A little bit alienating to anyone else in the room, though.

There is already a current product that is essentially this: https://moverio.epson.com

I bought a pair for experimentation as a viewfinder (about $1000). They proved the potential of that application, but suffered from some fit-adjustability issues and too small of an image. They could have been very good, however.

Yeah these have been around for almost a decade as cheap VR knockoffs for the unsuspecting, albeit I haven't seen the direct USB c output option before. Not until you take it home and turn it on do you realize the resolution of 1080p strapped to your eyes is a pixelated mess and the lack of even basic 3DOF makes these things uncomfortable for most people.

> Not until you take it home and turn it on do you realize the resolution of 1080p strapped to your eyes is a pixelated mess

Well, that sucks. Thanks for the clue to never purchase without a personal demonstration. I figured SD would be like that, but really hoped that 1080p would be satisfactory.

But doesn't this depend on pixel density? If the displays approach the pixel density of displays of high end smartphones, I'd probably be satisfied with that. Even though those pixels can be seen (at 10x magnification), at high enough pixel density, pixels can be ignored when focusing on the video content. I used to sit really close to CRT televisions, which have huge pixels, so there is bound to be some improvement from the way it was in the dark ages.

The Valve Index does 1440×1600 per eye and it probably needs to be at least double that before artifacts become hard to notice.

I am starting to see resolution doesn't matter. According to the specs, Valve Index has a very low average pixel density of about 13-15ppi. Criminal. Compare that to the high end iPhone display's pixel density of 128ppi. So if we could have ocular displays around 100ppi @720p, that would look a lot better than that of Valve Index displays at its seemingly impressive 1440x1600.

Valve index has a PPI of 598. What I assume you are talking about is PPD, or Pixel per Degree. Iphone doesn't have a fixed PPD since the distance to your eyes is not fixed.

An ocular 720p display at 100ppd would be be about an inch diagonally at 10cm from you eye balls. That's basically unuseable

Firstly a definition PPI: pixels in a single dimension / size in that dimension in inches.

A nice high end phones screen is like 300 PPI a 20 year old laptop screen is around 100. A nice desktop monitor is 160-240.

A calculator https://www.sven.de/dpi/

It's hard to discuss a monitor being 100ppi@720p because those 2 variables are related via a third variable size. A 720p screen would be 100dpi at about 14" and indeed that is a common size for crappy laptops. A tiny screen with that PPI strapped to your face would look like shit. For instance a 2" square screen would be 200px x 200px.

Resolution matters a lot.

A 20/20 or 1.0 visual acuity equals 1MOA(1/60deg).

A 20/20 equivalent 360 degrees panorama image is has a size of 21600x10800px minimum. For a VR headset with 120x120 degrees H/VFOV(178deg diagonally), you need a 7200x7200px panel. 2 x 7200px x 7200px x 24bpp x 144Hz = 360Gbps, or worth 4x 100GE LAG’d, or one PCIe 4.0 x23 link, or one DDR5 channel.

That’s a bit hard to put onto a face cheaply…

Could be done with fiber. I wonder if fixed pipeline optical processing could be used to transform the signal straight to display.

Our retina only needs ~8 megapixels - the same number of pixels as 4K/UHD (3840x2160 is 8,294,400 pixels).

High pixel density is only required at the middle of where our gaze is - the surrounding can be highly compressed because our eyes are not very sensitive. No pixels are required where our blind spot is, and our colour and light level sensitivities varies from fovea outwards, so there are other compression possibilities too.

If the display can "move" with your eyes (high pixel density only for fovea e.g. contact lens) then the display also only needs a limited number of pixels.

https://www.picturecorrect.com/what-is-the-resolution-of-the...

But the portion of the visual field that actually has good resolution is very tiny, so if you can get good eye tracking built in, you can cheat.

I’ve repeatedly tried to get into virtual dev work on my Quest 2 (1920x1832 per eye) but the screen density still isn’t there yet. Feels similar to trying to code on a classroom projector screen if that analogy makes any sense.

Quest Pro is hitting in October (supposedly) with 2160x2160 per eye but I still don’t think that gets density to where it needs to be for doing desktop work.

I think Apple has realized this, which is why every screen resolution leak I’ve read has it pegged at 4K per eye or more. But who knows when that thing will even get announced?

Regarding the Apple headset, I'll only believe it's real when Apple officially announces it — it's been bouncing around the rumour mills as "6-12 months away" for at least half a decade, with whatever specs sounded good that year.

(Also, I remember the initial rumours for the iPad; everyone I saw was saying it would run normal Mac OS X).

Yep, that’s why I personally don’t think the recent news about Apple trademarking “Reality” names has any bearing on the device’s imminent release.

That said, I do think we’re approaching an announcement since it was apparently demo’ed to the board earlier this year. I just want it to happen so I can get excited at a keynote only to go “holy moly, I’m not buying a first-gen Apple peripheral for $3000, Mister Tim Apple”.

HP Reverb G2 has a 2160x2160 full-matrix LCD. I would say it comes very close to being usable as a virtual monitor, but the virtual screen still needs to be quite large. Extrapolating from this and older generation headsets, I would guess 4K per eye should be enough for a true office experience.

On top of that, the headset itself must be comfortable to wear for sessions longer than 30 minutes. None of the current models are that comfortable.

How far away from 4k120 x 2 are current video cards?

4k120 x2 in $LATEST_AAA_GAME or in notepad?

The problem is that Apple, if they ever do release such a thing, will not provide an HDMI input to make the product versatile and useful. It'll be gimped by Apple's fear of connectivity, just like all of their iOS devices.

I dunno, I think with the recent EU legislation they’ll probably stick a USB C on it.

Doesn't matter. USB-C is not a video input. And if it were, Apple would still gimp it. I'm sure developers can't access the USB port on iPad Pros, just as they can't access the shitty Lightning port on anything else.

> USB C is not a video input.

Debatable. You can run monitors and these ar glasses over usb c and I’m pretty sure you can get hdmi to usb c adapters if that’s what you need.

> Apple would still gimp it. I’m sure developers can’t access the USB port on iPad Pros

This is semi-true. According to this stack overflow post you can get access if you’re part of their MFi program but that’s mainly aimed at corporates. You can try working around it using midi or raspberry pi if you’re doing personal tinkering https://apple.stackexchange.com/questions/13605/how-to-get-s...

I think you're talking video output, not input. Yes, there are USB-based video digitizers (for ingest), but you have to be running capture software on the host device. Very different from simply having a video input going to the screen.

You are entirely correct, my apologies.

More info on this stack overflow post for any future readers who stumble on this thread:

https://superuser.com/questions/1218848/hdmi-input-through-u...

Oh I fully expect to have to jump through several hoops to get a workable office environment on any Apple XR product.

I look at it like the newest MB Pros: if the hardware is way ahead of everyone else, I’ll hold my nose and put up with some amount of kludgy workarounds (eg: Parsec because no Bootcamp, etc).

I'm so glad Apple finally shitcanned their inexcusable "butterfly" keyboard and embarrassing emoji bar. Jony Ive's departure was the best thing for Apple in the post-Jobs era. The guy is a pompous hack.

The difference is that VR headsets have 120 or so degrees of viewing angle, whereas this is much narrower. This will look as sharp as an HD TV, probably.

There is the focal distance, which is the distance at which your eyes need to focus. I'm pretty near-sighted, so my focal distance is short.

There is also the focal angle, which is the angle your eyes have to be at to focus at a particular point which is a given distance away. As we evolved over millions of years, the focal angle corresponded to the focal distance the angle would get get closer and closer to zero (where your eyes are angled parallel to each other), the further away something was.

But with modern VR systems, they feed different images to each eye, and they give you different focal angles (to make things appear further away or closer together), but the focal distance doesn't change -- it's still that screen that is millimeters away from your eyeballs.

So, your visual systems get all kinds of screwed up. So far as I know, this problem is not something that can be corrected with any modern VR system.

AR is different, since you're layering one generated visual image with one screen that both eyeballs are looking at and that gets put on top of the actual real world image that you can see.

They are barking up the wrong tree, because as far as I know, no developer is paying attention to proprioception.

Not sure what you mean by this. VR headsets know exactly where your hands are and can infer most of the rest of your pose from head height, they rely pretty heavily on that for the immersion.

While position tracking is important, what I was talking about was the user's sense of proprioception. Seems to me that too much attention has been given to tuning the visual response to the user turning their head or looking up and down, or turning around, and little to no attention being given to attenuating the user's sense of proprioception, to reenforcing the feeling that the virtual body parts are an integral part of the user's actual real-world body.

Have you used any modern VR sets?

Depends. Are you attacking my credibility with an ad hominem fallacy? Just wondering. Because if so, it is only because my argument has confounded you, and if you can't beat that, maybe you can beat me.

But have you? Because I have and to me, within the confines of what is economically feasible (eg hand tracking but not leg tracking), they seem to spend a lot of effort on proprioception.

I think in a conversation about VR it's ok to ask whether youve actually used it.

> I think in a conversation about VR it's ok to ask whether youve actually used it.

I don't see how it's relevant. Nearly all VR insights came from those who never actually used it, namely because it didn't exist yet.

But, in fact, I have used VR quite extensively. I do own or have owned or borrowed or used or demoed at least 50 different models over the last 25 years, and my experience over the last decade has only ever reinforced this grievance, that VR hardware developers and content producers are paying little to no attention to the user's proprioception, and nothing I have ever seen, neither in my own experience, in depth reviews, documentation, nor your own statements has led me to believe otherwise. Though the component technology is getting better all the time, the VR development is no different than it was in 1996. It's as though developers are either entirely unaware that proprioception exists, or are aware and think it will take care of itself. And though it does sometimes, accidentally, it is not every time and always, and there is no great reason why it shouldn't, other than that developers apparently either haven't discovered it, or haven't discovered how.

Quid pro quo, now please give me your VR credentials so I know I'm not wasting my time. After all, once you can no longer speak to someone's argument, you think it is ok to personally attack them. So let's scrutinize your experience and knowledge in kind and see how much of an expert you really are, and maybe see if we can't get you to realize that when your ability to persuade fails, and you have no further ability to make valid argument and nothing more to offer, then you should really simply stop, leave others alone, and not try to bully or attempt to humiliate them with personal attack.

I simply asked if you had used any modern VR sets and you have started accusing me of 'personally attacking you'.

Nearly all VR insights came from those who never actually used it, namely because it didn't exist yet.

Everything gets invented by people who've never used it. But VR is an explicitly experiential medium. Again, I think in a VR discussion its perfectly fine to ask if someone has used it.

I have an Oculus Quest 2 (for my sins), when I first used it I was blown away although I find myself not bothering with it much now, there doesnt seem to be much to draw me back there, which makes me think Meta are heading for trouble. But thats an aside.

I'm confused about your argument and I think perhaps by proprioception you mean something different to what I'm thinking?

Because my experience with the Quest is that the device knows exactly where my hands are, their exact orientation and even what my fingers are doing. And it puts a lot of effort into making my 'hands' part of the experience by showing them to me in VR. This to me seems to be leaning heavily on my innate sense of proprioception to maintain the VR illusion. True, it doesn't know where the rest of me is. But the hands are the most important part, hence it seems that within the confines of what is economically feasible (eg hand tracking but not leg tracking), the developers of the Quest have spent a lot of effort on proprioception.

But you maintain that VR developers are entirely unaware of proprioception. So I think you are talking about proprioception in a deeper sense? Or in a sense that has not occurred to me? I'd be interested to know exactly what you mean.

> I simply asked if you

My point was lost. If your argument is valid, it will avoid any mention of your opponent and their personal circumstances and focus only on what was said. Otherwise, you are constructing a fallacious argument known as the ad hominem fallacy.

Regarding proprioception, you are likely very aware of most of your senses, such as sight, smell, taste, etc. Proprioception is a sense precisely the same as the sense of sight is a sense, and it is merely the perception and awareness of the position and movement of the body and it's parts. Without looking to see, you know where your nose is, you know where your hands are, you know if your feet are, say, pointed towards or away from each other or anywhere in between. This is the sense of proprioception. When you become experienced at driving a vehicle, you will have good sense of where the perimeter of that vehicle is, including the edges of a trailer if you are towing. This is an extension of proprioception, the vehicle more or less becomes a part of your body, and you know its edges without actually constantly checking to see where they are all the time.

In VR, what you see and hear and sometimes feel should give the sense of proprioception that what you see yourself manipulating, whatever it happens to be, should feel as though it is a part of you. There are tricks that developers can use to reinforce this sense of proprioception, but beyond seeing a moving limb in feedback to your own actual limb movement, I have never seen any development of proprioception beyond this.

There is a body illusion known as the wooden hand or wooden arm illusion. Your arm is placed under a table so that you can't see it, and a wooden beam (say, a short 4x4) is placed on the table so that you can see it. The assistant has two feathers, one in each hand, but you can only see one. Simultaneously the assistant strokes the wooden beam and your hidden arm with a feather in the same motion, and within seconds, your sense of proprioception (shockingly) fools you into believing that the wooden beam is your arm.

This is just an example of how sensory reinforcement is able to fool your sense of proprioception into rigorously believing that what you see is you. I have rarely, if ever, very deeply believed, so far as to the suspension of disbelief, that the elements that I've manipulated in VR have ever become an integral part of me, proprioceptively. Developers are ignoring proprioception, and all it takes is to clue them in somehow ("hey! Have you guys heard of proprioception? Look into it!"), and they will do the rest, which will take research, understanding and then changes to the product and content to constantly reinforce the illusion, and when they do, every VR experience will truly be immersive and not just said to be so in lip service by marketing materials.

Just because one subjectively likes something does not mean that it is objectively any good. Without addressing proprioception, no matter how advanced the underlying hardware technology gets in resolution or surround sound, unless proprioception is placed as the primary concern, VR will continue to kind of suck and never become mainstream. Everyone should have VR hardware, it should be everywhere in education, in commerce and in industry, and right now it is nowhere but in the rare gamers' inventory.