How to Compare Images Fairly (2017) | Hacker News Reader

How to Compare Images Fairly (2017)

kornel.ski

30 pointsjsnell7 comments

The "Compare at realistic quality" section is a bit off the mark itself. 1 KB images are a realistic use case be it placeholders to low fidelity background images to cheap ass mass thumbnailing. At the same time lossless or near lossless images are also a use case be it for data preservation or a full quality image view. As such it's not that there is an "unrealistic quality" setting to avoid testing it's that when you test you either have to frame your results in context of a specific use case or do a comparison across the spectrum to get an idea which codecs have which strengths and weaknesses. This is basically an extension of the "Test on many images" section at the end where aspects include not just content of the source image but intent of use for that content.

"1KB" is not a quality. The article doesn't tell you not to use thumbnails, but rather compare images only at qualities you'd actually use.

So if you're interested finding a codec that is best for low-quality thumbnails in particular, then comparing low-quality thumbnail images is the way to do it.

But that section exists because there are many comparisons on the net where people extrapolated cheap-ass quality test to everything. It's not correct to find the best codec for cheap-ass thumbnails and conclude it will be good at other image types too.

Consider a simplified case of the classic JPEG vs "JPEG-with-blur-applied" codec. If you compare both at cheap-ass quality, then the JPEG-with-blur will look better: less blocky, without ringing artifacts. But the thing that makes it good at low quality obviously backfires at any higher quality, because it can't preserve details smaller than the blur radius. Modern codecs actually have features that behave like this (in-loop deblocking, intra prediction), so you need to be mindful if you're after quality ranges that are made better or worse by blurring.

> "1KB" is not a quality.

1 KB is indeed not a quality, it's the target size of the "fallen apart" example images in that section.

> The article doesn't tell you not to use thumbnails, but rather compare images only at qualities you'd actually use.

> So if you're interested finding a codec that is best for low-quality thumbnails in particular, then comparing low-quality thumbnail images is the way to do it.

> But that section exists because there are many comparisons on the net where people extrapolated cheap-ass quality test to everything. It's not correct to find the best codec for cheap-ass thumbnails and conclude it will be good at other image types too.

Again it's not really anything to do with "compare at a realistic quality" because there is no one quality that corresponds to real world use cases with which one can compare at. The problem comes from people coming to conclusions/claims with extrapolated data not that the single quality they picked happened to be a certain value e.g. low. Had they singularly picked mid or high quality and extrapolated they'd still have made inaccurate conclusions by extrapolating. The only appropriate ways to handle it are as stated, either frame your results in context of a specific use case or do a comparison across the spectrum instead of picking a quality.

> Consider a simplified case of the classic JPEG vs "JPEG-with-blur-applied" codec. If you compare both at cheap-ass quality, then the JPEG-with-blur will look better: less blocky, without ringing artifacts. But the thing that makes it good at low quality obviously backfires at any higher quality, because it can't preserve details smaller than the blur radius. Modern codecs actually have features that behave like this (in-loop deblocking, intra prediction), so you need to be mindful if you're after quality ranges that are made better or worse by blurring.

Better to to not be mindful of things like this and only speak to the result data. It doesn't particularly matter how the codec gets better results just that the scoring algorithm says it did. Of course bearing in mind to not extrapolate past your data (in any direction) as mentioned above otherwise you're back to talking about your assumptions not your testing.

I suspect you're misreading "realistic quality" as "good quality". I mean "realistic" as opposite of "unrealistic", i.e. a quality level that you'd never want to use in practice. Don't benchmark with a quality level that is stupid/useless/obviously inappropriate for your usage of images that you actually want. It's almost a tautology, so I'm really surprised it's controversial.

My disagreement is that there is such a thing as a realistic or unrealistic quality one could use for images in the first place as if there is some very limited SSIM range one isn't supposed to want to use image codecs outside of that would result in irrelevant benchmark data.

The blame being attributed to this actually belongs to extrapolating from a single test case and assuming it implies something about all the test cases as opposed to having run an invalid test case. Particularly when referring to the cases of people assuming how a codec performs at lowest quality implies something about the higher quality comparisons. The issue here was never about the lowest quality options being unrealistic it's about assuming results from one test were directly correlated to what another would show

If I were to replace the section with where my thoughts on it are it'd probably be titled something along the lines of "compare across a full spectrum of quality" and detail why you want to avoid extrapolation from a small range of test scenarios instead of avoiding some range completely.

So I think your argument is that a single quality level is a single datapoint, and therefore not enough to say "codec A is better than codec B on average".

That is technically true, but I'm not even suggesting to attempt to draw such conclusion, because (apart from an easy case of a Pareto improvement) such total order may not even exist. It can be both true that codec A>B and codec B>A at the same time. Therefore no matter how much data you throw at such comprehensive comparison, and how meticulously you aggregate it, it's going to be false/misleading in some way.

There can be a "best codec for smushed thumbnails" and a "best codec for sharp icons" and a "best codec for archiving scanned film", but if you average over these categories, the "overall best codec" may be inferior at all of these.

So while comparing images at one quality level doesn't generalize to "codec A is overall better than codec B", it does give you a more accurate "codec A is better than codec B for the quality I'm planning to use".

1 KB might be a realistic size for a low fidelity background or thumbnail, but 0.01 bytes/pixel or 0.06 bytes/pixel isn't a reasonable compression target. In other words, when making a thumbnail you should probably be downsampling the input rather than just feeding a 320x203 image into your compressor and turning down the quality setting until it fits under 1 KB.

Reply on news.ycombinator.com