Hacker News Reader

Comment by jcl | Hacker News Reader

It looks like they are using a "multi-scale structural similarity" (MS-SIM) metric as a proxy for how well a human will think a compressed image reproduces the original image. Both the JPEG and the RNN images are compressed to the same MS-SIM metric, which in this case makes the RNN image take 25% fewer bytes.

Part of the reason JPEG looks better in this case is that they have chosen to feature a detail- and noise-rich portion of the image, which favors JPEG... High JPEG compression has a sharpening effect around edges, including "ringing" which hides well in noise. However, if you view the full-sized comparison of the dog images, you can see that JPEG has essentially no detail in the darker portions of the image. Presumably both the MS-SIM metric and humans would judge RNN better for these portions.

View on HN

Thanks for describing MS-SIM - I've always been concerned about using L2 or similar metrics for image similarity. Is there any work on learning an image similarity metric? I would think if you Mechanical Turk'd a bunch of A/B images (with or without perturbations of various sorts or even totally different images) and asked the user to score the similarity you should be able to learn a pretty interesting scoring.

Netflix did that for video: http://techblog.netflix.com/2016/06/toward-practical-percept...

Thank you!

>High JPEG compression has a sharpening effect around edges, including "ringing" which hides well in noise.

Ringing is the opposite of a sharpening effect, no? JPEG essentially acts as a low-pass filter over the DCT tiles of the image, so you get ringing because you eliminate the high-frequency components that make the edge sharp and you're left with a wavelet that extends past the sharp edge and isn't cancelled out the way it used to be.

Nope, ringing is the effect of a high-pass (or band-pass) filter - it's the opposite of smoothing.

JPEG looses information by quantization, not low-passing. It is true that small high frequency coefficients will get rounded to 0 during quantization, and this does save quite a few bits. But the ringing is just from being imprecise about each DCT coefficient, coding a value that is either too large or too small.

I don't think it makes sense to say that ringing is (necessarily) an effect of high-pass, low-pass, or band-pass filters. Any second-order system (that is undamped) could exhibit ringing. If I concatenate a low-pass filter with one of these ringing filters, I could get a ringing and low-pass filter

Remember that a brickwall band pass filter (low, high, or anywhere else) is equivalent to convolution with this https://www.dsprelated.com/josimages_new/mdft/img1768.png in the spatial domain. Any time quantization is high enough for coefficients in a given band to become zero while the surrounding ones remain nonzero it will introduce at least some level of ringing.

wyager accurately described why in terms of the DCT though -- cosines naturally "ring" and the only way a non-sinusoid can be represented is through the combination of many other terms, so any time a coefficient contributing to this shape is eliminated, the structure "decays" so to speak and the underlying sinusoids become part of the reconstruction.

The NN images, for sure, have a problem reproducing sharp edges/have less high frequency content. Just look at the grass.

It appears to be a severely flawed metric.

By severely flawed metric, I meant,

> Presumably both the MS-SIM metric and humans would judge RNN better for these portions.

doesn't remotely hold for detailed/high contrast. If I can see a huge difference, and vastly prefer one over the other, the metric is not useful by its definition. Please don't get emotional about figures of merit.

I'm guessing a genuinely useful figure of merit would have put both file sizes a bit closer together, and the NN would have shined, especially since it works so well in low contrast areas.

Reply on news.ycombinator.com