Hacker News Reader

Comment by scoopertrooper | Hacker News Reader

I think this could be used as a great basis for next generation video conferencing. Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses.

Most current networks can handle video conferencing (relatively) well. However, if 8k video or something like Light Field displays ever catch on, then we might end up needing orders of magnitude more bandwidth to drive video conferencing, so this might be an enabling technology.

View on HN

Might not be that simple. In a recent demo from Unreal Engine they said that the facial animation data is the main bottleneck to shipping titles with more realistic avatars. Their minute-long demo took many TBs to store the motion data alone.

My layperson understanding makes this baffling to me. Isn't motion inherently analog, and haven't we've developed high performing analog compression already? Motion seems significantly more amenable to lossy compression as well, and I could imagine a use-case for ML-based decompression to fill in between the keyframes.

You also have to consider that faces are one of the things that we have the strongest perception of, with lots of our neurons dedicated to the task, so when you get things wrong it's far more noticeable than many other bodily animations would be.

One of my favorite “weird” conjectures is that the existence of the uncanny valley implies that at some point it was evolutionarily advantageous for us to recognize something that looked human, but wasn’t… and to be afraid of it.

neanderthals?

Think also about face-like things we perceive because of pareidolia but might in fact be dangerous. One part of our brain is opportunistic about finding patterns while another is a sanity check on those patterns telling us not to just trust them outright.

Is there any further reading on this topic in particular that is not leading to wild conspiracy theories?

I had the same reaction (note: I actually got things mixed up, the one in question is the Unity "Enemies" demo [1], not Unreal), but if you consider the amount of detail - the saliva 'sticking' to the corners of the mouth, the 40+ muscles of the face, lips, eyelids, the iris, etc - it makes sense that it would be a ton of data. IIRC they are working on some kind of ML-powered compression so that this becomes actually possible to ship (the demo is not available to download, presumably it's too big and would barely run on a 3090TI).

[1] https://www.youtube.com/watch?v=eXYUNrgqWUU

Facial animation is a real issue. This is facial mapping, which is an easier to solve problem, in that throwing artists at the problem is a solution.

I guess, would watching an animated head avatar in 8k or a Light Field display be a better experience than watching lower resolution video?

I wonder whether the experience would be too undermined by artifacts from e.g. your coworker's cat, for which the system doesn't have a model, wanders into frame partway through a call, and suddenly the system must dynamically either (a) 'downgrade' to just video (b) be able to create a new model for an arbitrary new thing on the fly (c) create some hybrid of partial video data with part viewer-rendered animation or (d) your coworker looks crazy interacting with something the system hides from you entirely.

Or, at least, the faces that aren't actively speaking could be replaced by avatars. It will also help keep a clean image in case stuff is going on the the background, etc.

> Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses.

What about latency though? Would that improve?

there's another thread about just this, but very different approach. reduces bandwidth a lot specifically for video conferencing

Reply on news.ycombinator.com