Hacker News Reader

Comment by varispeed | Hacker News Reader

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

View on HN

Video is not real either. The perspective is different and the colors are wrong.

It is even less real after it went through lossy compression. The whole point of a lossy compression is to remove details that you perceive as unimportant. For example, leaves on a tree may look like a greeny mess, but that's fine, from afar, you don't make a difference.

Using neural networks for compression is far from being a new concept, and the result is not more or less real than any other technique. It is just that Nvidia implementation is really good at keeping the most important details in a small size.

If you want a more "real" image, you can just use the AI as a predictor and use the remaining bits to encode the difference, like in a traditional MPEG-style codec.

I understand what parent means, usually processing was too linear to alter the meaning of what you see, it's distorded, altered, maybe filtered and enhanced but in general, it's the same. With ml/dl you can really be sent .. whatever.

In some ways, of course, I agree, but as some other commenters have pointed out, this is essentially a "deepfake" of yourself...in principle there's no restriction on what one could use as their "avatar." As much was demonstrated in the "Different Output Styles" slide.

So I'd have a hard time saying this is not dramatically less "real" than some lossy compression technique. Is there some way to formalize "realness?" Maybe it would be inversely proportional to the hardness of manipulating the medium as a user..?

Maybe simply using an objective video quality measurement against the original picture would do the trick. Something like PSNR or SSIM. A "deepfake" is likely to score low on such a score that doesn't depend on high level visual perception.

Also know that you can "deepfake" yourself using a traditional video encoder, just change the keyframe to someone else's face. Of course, it will look broken and totally unconvincing but because of motion compensation, you can sort of map the movement of your face on someone else's face.

The technique in the paper simply has way better motion compensation, so good that it still works if you change the keyframe. Traditional video compression algorithms don't work like that because they are not just for talking faces and can't use such advanced techniques for performance and ease of implementation reasons.

I think the point is that for very low bandwidth, the "deepfake" version will score higher than the heavily compressed video stream for PSNR.

The distinction that makes ai compression creepy is that it can use prior knowledge of what a face is.

Traditional compression has no high level knowledge of what a video call looks like, the algorithms are about patterns in how pixels change in time and space, so the artifacts from the algorithm are pixel-based effects (blockiness, blurriness, etc).

A neural net that has been trained on a million faces and is setup to draw a face may draw a perfectly clear image of a face on low bandwidth...but when it doesn't have bandwidth to be accurate, it doesn't blur things, it fills in details learned from looking at strangers who aren't on the call.

You're sitting at your computer, you're watching your colleagues are discussing something in detail, slowly you start to notice - your two black colleagues are starting to literally look the same. That's weird, are you slowly getting more racist? oh right, your bandwith dropped and your AI upscaler has decided to upscale all your black colleagues up to some stereotype it got trained with.

I'm afraid I suspect this isn't that far fetched.

come on man there's a difference between a GAN hallucinating 90 % of the image and a very predictable compression algorithm where both parties understand what's going on

"the result is not more or less real than any other technique"

I disagree. If real / not real is binary, sure, it's not real. But if we allow "real" to be a range of values, it is less real.

That's how your brain works as well.

Adding a secondary, uncontrolled layer of perception is definitely not the same thing, but in general, what you "see" is barely "real".

If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally sends both things, then the receiver should be able to always piece together a 100% reproduction of the actual video feed on my end. The transmitter can always predict exactly what the receiver will decode, so the correct amount of data to send is the amount of data that makes the receiver see what I want. If that means I send 0.1mbps key point data and 1mbps diff pixels then that’s what I want to send.

Is that how this works? Because it should be how it works...

Not sure if this is a viable way of compressing the video stream or if a transmission of the “diff” would be too costly. If this method would give 1/2 the original bandwidth I’d think it’s more impressive than cutting 3 orders of magnitude via “avatars”.

I suspect that this isn't feasible. Otherwise lossy video codecs wouldn't exist (or would be very niche). Because if you sent a perfect diff you would essentially have lossless compression again.

So either your idea will revolutionize video compression or the "diff" would bring you back to the ballpark of lossless codecs.

It is a lossless codec yes, but I has a “base” image for the key frames (which are the most expensive) which can be transmitted as the key points. The question is how much that actually helps the end result vs how much it costs. As I said it would likely kill at lest nearly all the gains, but again - a small improvement in the state of the art for “normal” looking video would be more impressive than three orders of magnitude vs sending instructions to my head puppet.

You’re thinking along the right lines, but the challenge is that a raw diff will have the same number of pixels as the raw image, so no compression in bandwidth. So, how do we represent the diff/residue also with fewer numbers? At that point it’s the same as choosing better parameters within some clever encoding (be it pre-designed like JPEG or H.264 or learned via ML).

I was thinking that subtracting the predicted image would give an image that has more zeroes and compresses better (much like dct+quantization for jpeg). After all, any time the neural network would predict an area of the image almost exactly, it can be omitted from the diff stream completely too.

The diff will be low norm (for some suitable norm), but it needn’t be sparse in pixel space. It could be sparse in some other spaces (Eg: wavelet basis), but that’s right where the challenge lies — finding a basis where the data (and/or the residue) to be transmitted is sparse.

It could be bad for telemedicine, since the NN may gloss over some "weird" pathological features not found in its training set. The current setup involving an initial image and keypoints should be fine though, as long as the initial image (keyframe?) is not messed with and is transmitted frequently enough.

your brain does lots of preprocessing to create a stable view of "reality", glossing over blind spots, retinal blood flow, etc. in a way we are always hallucinating, and it seems natural that computers should learn what we are actually paying attention to in video, and focus on that

Yes! E.g. AAA games render the entire 4k screen with the highest graphics, but only a small spot hits the part of retina that can perceive jagged edges or lighting details.

The raw sensor data for a digital camera looks something like this https://commons.wikimedia.org/wiki/File:Normal_and_Bayer_Fil...

In addition to the colour grading and interpolation, lens defects might be 'fixed' before you ever get data on disk.

ML is just really good interpolation.

any lossy video-compression algorithm face the same challenge. what you are seeing is artificial and is constructed by the algorithm to minimize the perceptual difference between the real and the constructed video feed.

You'd get used to it very quickly. Like you got used to TV and film, which differ substantially from an in-person view in many ways.

Reply on news.ycombinator.com