Hacker News Reader

Comment by endisneigh | Hacker News Reader

What's the necessary resolution in each eye needed to be "retina"? Once we reach that will it be the end of the "monitor era"? A lot of interesting things can happen once you no longer need monitors. Though sharing with others will still be a practical limitation so I doubt they'll go anywhere for now.

Once you no longer have monitors I think there will have to be a UI and UX shift around how you interact with things. Windows made sense because you had a fixed set of boundaries. If the boundaries no longer exist it probably doesn't make sense to have windows in the traditional sense anymore. You could have an infinite set of context sensitive menus that appear as necessary.

View on HN

A field of view about 180 degrees wide, 135 degrees high, 1 arc minute in maximum pixel size, and updated 100 times per second… twice, to account for full stereography, and using 48 bits of color.

180 * 135 * 60 * 60 * 100 * 48 = 419904000000,

420 billion bits per second, without compression.

Keep in mind that the 90FPS benchmark is merely required to achieve "presence," which is the phenomenon of the brain accepting the virtual reality as reality. Higher framerates are always beneficial (as this also reduces latency).

> 420Gbps

I assume foveated compression (a pet idea of mine that I have no doubt someone else also has) is probably the only way to realistically deliver the stream to the HMD (assuming that some rendering pipeline that can deliver the uncompressed stream exists).

You also have to be concerned about micro-saccades, which may increase the effective resolution of the eye.

Right, 420Gb/s is the worst-case number. If you can track the fovea accurately, at 100 Hz you need to keep at least a 9 degree circle at full resolution -- more likely an 18 degree circle -- and even so, there's no mechanical system likely that's going to keep up with that motion, so you need a display screen at full resolution across the entire surface, even if you're only painting full resolution across a small sector each frame.

Also also note: there's head motion to track and events to respond to. If you're playing a movie, that's pretty much your best case because you only have to pan and tilt a stream of things you already know about. Playing a game is medium bad, because you can predict the range of possible events. Doing AR mediation is worst, because you can't predict the rest of the universe.

When you say "foveated compression" do you mean >Apple will liberally use an already-known VR technique that involves using eye-tracking to render objects in the user's periphery at a lower fidelity than what the user is focusing on. ?

Interesting recent discussions on Foveated Rendering https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

No foveated compression is not foveated rendering. It is using the same general idea for a different area of the problem: a higher bitrate (on the wire, or across the wireless link) for in-fovea.

Due to latency, you can't render the fovea area only; you need to cover quite a bit more in case the eye moves between submitting the command buffer and the final image being ready.

Compression can happen much later in the process, where the latency is much more predictable or even fixed. This is useful because massive bandwidth is easiest along traces, harder along wires and hardest wirelessly.

What's the resolution in terms of a screen? Like height by width in pixels? Sorry if that's obvious from your comment, coming at this with just basic consumer knowledge about displays.

The eye doesn't care about pixels, the eye cares about the smallest detail that it can distinguish. If you specify a particular distance you can figure out how small the pixels have to be at that range so that you can't see them. It's around one half of an arc-minute, which is to say, 1/120th of a degree.

https://en.wikipedia.org/wiki/Visual_acuity is a really useful overview.

Now, we know that human eyes can only cover a little less than 180 degrees horizontally, and about 135 degrees vertically, so we could multiply each of those by 120 and get:

180 * 120 = 21600 across by 135 *120 = 16200 down. That's a screen where every pixel is guaranteed not to be distinguishable from the ones around it. If you had this on a hemispherical screen 10cm from your eye, your square pixels need to be no larger than 0.0145 mm on a diagonal, or if you prefer have a dot pitch of 0.0145 mm. Current state of commercial availability smartphone screens have dot pitches of about 0.045 mm, about 3 times too big.

Does that help?

Probably can cut that down further due to :

* People's vision getting worse with age

* Each OLED pixel consists of 4 subpixels, so may only need to reach that acuity at sub pixel resolution

* Acuity drops and eyestrain increases at the edges of vision so likely only need maybe 75% of our range U/D L/R at the highest resolution, and the outer edges can have much lower resolution

Other hacks for foveated rendering include rendering peripheral vision at lower resolution and monochromatic, and not rendering in the blind spot, since our brains will just fill in those areas. May not be worth the complexity, but there's a lot of hacks that could take advantage of our visual system's quirks. I imagine ray tracing makes these hacks even easier to implement.

From the logic of the GP, that would be:

Height: 135 * 60 (135 degrees at 60 pixels per degree)

Width: 180 * 60

So 87 megapixels, or a 10.8K display with a 4:3 aspect ratio

In the future there may be an integrated display technology that physically cannot display an arbitrary combination of pixels at high speed, but can only do so for the space of "images which humans are likely to see" which I suspect is an unimaginably smaller possibility space.

I suppose something like a neural representation of a scene that is directly rendered on the underlying hardware, without first being converted to a raw bit stream.

This is an interesting return to the way that CRTs worked. They were technically 0-dimensional displays (a single beam changing in intensity, which was moved by a non-programmable magnet around the tube)

In our modern future display, we would similarly blast data to only where it is needed at an instant in time.

Foveated imaging can dramatically decrease the required bandwidth. We have about 6 million cone cells and about 1 million of them is in the fovea centralis. Rods are more numerous, but they are mostly for low-light vision.

Depends on the field of view angle you want to achive, but 8K is unlikely to be enough.

Back in 2015 Michael Abrash said:

In order for it to have retinal resolution across a field of view of 180 degrees it would have to have something on the order of 16K by 16K resolution [per eye]. [1]

[1] https://youtu.be/UDu-cnXI8E8?t=4232

> Apple will liberally use an already-known VR technique that involves using eye-tracking to render objects in the user's periphery at a lower fidelity than what the user is focusing on.

Should be enough if they don't have to render that full view, but only what the retina is focusing on I would think.

Which suggest and interesting solution: a very small high res display surrounded by much lower res areas that can move around in such a way that it always sits right where the eye is looking.

I think it would be easier, more reliable, and more energy efficient to wait for microled displays, which seem to be right around the corner.

I get the impression that they might be available by the end of the year or next year, and might be something Apple has access to?

that solves the gpu problem of high-resolution rendering and the bandwidth issue, but is the density of pixels in the display enough to fool the eye?

I feel like the windows are there to create context for the human brain, not because they resemble the shape of the screen. These context may look a little different to windows, but I don't think they will be radically different in principle and will still maintain a distinct resemblance to the physical world. That's what our brains are moulded to best interface with after all.

About 3000ppi

Example headset that used a small display with this pixel density.

https://arstechnica.com/gadgets/2019/02/retina-resolution-he...

Reply on news.ycombinator.com