Hacker News Reader

Comment by dotancohen | Hacker News Reader

The fine article shows the low-res input and high-res output photos, but conspicuously does not show a high-res original from whence the low-res input was derived.

Without comparing a high-res original photograph to the high-res output photograph, we do not know if this fine technique is capable of producing nice-looking high-res imagery, or if it is capable of reproducing how an image of the subject would have looked like had it been taken in higher resolution.

In other words, does the output of the technique match the actual object in the photograph?

View on HN

That is indeed a shortcoming of this article in my opinion as well. If you want a comparison to the original high res photos, there are some examples in the original paper for SR3: https://arxiv.org/pdf/2104.07636.pdf Have not had a look at the CDM paper.

What strikes me is that the jaguar (?) got "kind" eyes with SR3, and "mean" eyes with the other algoritms, including the original!

Fig. 9 of this paper is really interesting if you zoom in. It looks like if the model was not trained on the appropriate class label, it just goes completely off the rails. As previous commenters have noted, I would be very hesitant to use this for anything analytical, or where you are looking for something unexpected. For faces though, this is amazing.

> where you are looking for something unexpected

I'd be extremely sceptical of its use on medical images for this reason.

Of course it does not. The model generates "believable" images not exact ones.

The point is, it's helpful if the reader can evaluate just how faithful the transformation is vis-a-vis the original.

Helpful, interesting, yes. BUT not a real critique on this paper which specifically does not claim anything regarding similarity with the original.

This happens a lot with junior peer reviewers, they focus more on what the paper isn't or could be instead of what's actually there.

Sure, if you are evaluating at whether the claims are accurate on that level of precision, similarity may not be relevant.

However for the rest of us, evaluating whether the work is relevant in the surrounding world really is important. And for that, resemblance to originals is critical in some use cases. I.e. we're evaluating a fit for specific use cases.

Do you believe that is not a useful discussion?

This isn't obvious to regular people, though.

I can recall seeing a conspiracy get traction on Twitter a few months back, where it was claimed that a photograph of a famous person was actually a body double. Someone used an ML upscaler to "enhance" the image, and their followers began scrutinizing the result: "The teeth are different!", "The nose shape is wrong!", "It's not the same person!"

There is an app called "face app" or whatever that already did pretty good job upscaling people's face using state-of-the-art AI upscaling.

The result is impressive, but the moment you started to use it on someone you're actually familiar, it becomes weird very quickly for the obvious reasons. The teech, for example, are never right.

This kind of "believable but not truthful" results are rampant in all these machine-learning based tools. It's not very harmful in case of upscaling a few photos I guess, but I've been bitten by it in an acclaimed translation service called DeepL. I use it to translate Japanese to English frequently, and have found that it often (nontrivially) made up sentences that don't exist in the original paragraphs, sometimes have the opposite meanings, or totally ignore part of the text to make the result "more fluent". And unlike traditional translation tools, they are very hard to notice if you know nothing about the original language. I have to from time to time use some more "primitive" translation tools, and compare the results side by side, to avoid such issues. It's frustrating.

Then why is it mentioned that one of the potential uses are in medical imagery?

Not to mention what happens when some good-intending but ill-informed agency release an enhanced photo of a suspect on the run.

> Not to mention what happens when some good-intending but ill-informed agency release an enhanced photo of a suspect on the run.

The more interesting situation is when such "enhanced photos" are used in prosecutions. I suspect that ML "forensics" techniques and questionable expert witnesses will be used to elevate hunches to guilty verdicts and raise (false) conviction rates.

I think it would be possible by any defense attorney to show how bad a job it does at reproducing the original by showing hi rez images (actual) vs AI enhance upscales. Shouldn't be hard to demonstrate "reasonable doubt"

If you follow the referenced page[0], you'll find examples[1] showing the high-res originals.

[0] https://iterative-refinement.github.io/

[1] https://iterative-refinement.github.io/images/super_res_exam...

> In other words, does the output of the technique match the actual object in the photograph?

Probably a lot in some cases and a little bit in most others. I wonder how long before this gets used in court by an incompetent prosecutor.

How do you expect an algorithm to create information out of nowhere to fill in these details exactly as they were in the source photo?

If this AI can't produce upscaled photos that are close to original photos, the application of this tech is severely limited. There's not going to be CSI like "enhance" moment in real life like the article claims.

You really don't see any use for this if it can't create information? CSI "enhance" is, and always has been impossible.

It does create information though, just imaginary/wrong one.

No, it doesn't. A piece of code n bits long can only have a maximum of n bits of information in it; you can't get more than n bits of information out of it. It can't create information out of nothing.

I'm not sure what you mean, but that is pretty much what generative neural systems do. If by information you mean data that correspond to reality? Than sure. I guess you can say that the data is a correlated to reality by a certain probability, and the better the model the higher the probability?

Information in the sense of information theory. You can't squeeze n+1 bits out of n bits of information.

> You can't squeeze n+1 bits out of n bits of information.

Sure, in the extractive sense. But in the generative sense you can.

The fact that the extra information is 'synthetic' and not 'natural' doesn't mean that it isn't extra info, just that it may or may not (probably not) correspond with ground truth.

Another way to think of it is that super-resolution is effectively a (possibly benign) man-in-the-middle. If what you're concerned about is the information flowing from Alice to Bob, Eve isn't adding any info, and may in fact be drowning the signal in more noise. But you can also see it as Eve communicating more info to Bob than Alice is to Eve. Whether what Eve is adding should be considered noise or signal is highly context dependent.

Okay, then where does the information come from? A deterministic algorithm can't generate information where none exists.

> A deterministic algorithm can't generate information where none exists.

Right. All the processes being used for these purposes are stochastic.

Edit: Actually they are deterministic, in the same way a pseudo-random-number-generator is, and typically rely on a 'seed' that would have to come from a random source to be non-deterministic, and doesn't, so users get to have pseudo-random but reproducible results. But that's really getting into the weeds.

Have you heard of lossy compression? Are the application of lossy compression "severely limited"?

Meh! I'm sure there are use cases for this format but it ain't what this article claims it to be.

Very simple, you _predict_. If you can tell the low res image is of a person's face, the likelihood of possibilities when filling in the blanks in the high res image are significantly reduced.

Reply on news.ycombinator.com