Gamma error in picture scaling | Hacker News Reader

Gamma error in picture scaling

ericbrasseur.org

55 pointsrubatuga33 comments

I'm not an expert, but I've dealt with my share of this working in film. Color problems in image processing are everywhere. Pre-multiplication and Alpha channels are another huge one. A lot of professionals don't even understand it and will brute-force compensate for problems instead of understanding the problem. Here's a posting from this week [1] from professionals who care about getting this stuff right trying to convert a tiff to jpeg without surprise colors shifts.

A summary is that you need to do almost all image modifications in a linear color space. So you first need to back out any color tweaks made (for aesthetic or technical reasons), modify the image, then re-apply any tweaks.

Unfortunately, image files often aren't tagged with their color space and might use a specific color space out of convention--but they could use something else. Also, because it's way harder to do the right thing, a lot of software just does the modifications ignoring color space. Just think about how much slower and more memory a web page full of thumbnails would take to draw if you added in color space transformations? For awhile, a lot of browsers added support but left them off by default (imagine if you designed your web page to look right in a broken color space and it was suddenly much slower and now looked wrong). But a lot of software is slowly catching up. Unfortunately, legacy software (like Photoshop) and legacy file formats might never change.

[1] http://lists.openimageio.org/pipermail/oiio-dev-openimageio....

FYI You can get gamma correct resizing (and other operations) in recent versions of photoshop by first switching the image to 32-bit colour depth mode.

I had a really long reply typed up with some specific workflow problems. But you can see people complain elsewhere in this thread about how Adobe handles stuff like this.

The basic problem is that the color space for blending is an application setting (can't be specified in the file) and higher bit depths will double or triple your file size.

You could do a rough approximation of gamma correction with squares and square roots, which wouldn't add to the processing time like a full sRGB conversion would.

Going from 8-bit sRGB channels to 32-bit IEEE float channels is a simple lookup in a table with 256 values.

Going the other way is only slightly harder. Scale to the range 0 ... 4095, properly round the float to an integer, verify the range or mask with 0xfff to protect against NaN and Inf, and then to a table lookup in a table with 4096 values.

Tux Paint contains GPL code for it. It's been there for more than a decade. All you GIMP and Photoshop users having trouble with this should have been using Tux Paint. :-)

Lookup tables are cache killers on modern CPUs. Arithmetic is often faster.

It's 5120 bytes for the both of them, the access is not at all random (due to color popularity), and the real cache killer is the image itself. You're dealing with a few megabytes for the image.

Oh, the lookup tables are cached just fine. It's just that they crowd out something else.

That "something else" is already crowded out by running through a multi-megabyte image.

At least the instruction cache is barely affected, which would not be the case for a math library function.

You're not running a multi MB image but a small block thereof. Not that 5-6KiB LUT matters, but the extra conditional multiply ops or power or logarithm function evaluations do. Latter are need a few MACs for a good approximation (at least 12 bit precise, preferably 16 bit - like 5th order polynomial), former have masks, worse for SIMD.

I had this idea a long time ago but never actually tested it. Now I have: http://marksblog.com/gamma-resize/. The square/sqrt method gets at least 80% of the quality but should be quick to compute on a modern processor.

sRGB gamma encode/decode is a natively provided hardware feature on GPUs, and is plenty fast. There’s really no excuse in 2019.

The irony is that all accelerated hardware resizers available on ARM SoCs that I know are also not gamma aware.

This is why downscaled videos feel to have more contrast, or solarised due to bruteforce compensation on most Android devices.

Those LUT or power or logarithm functions are not free... linear light scaling is about 2x slower in practice. Not that it matters until you get to 8K.

That's an interesting point. The GPU should do the resizing too if they're using it.

I first saw this post in 2009 or so, and it was eye-opening. I found it while googling for why colours shifted when I resized images, as back then there wasn't much awareness of this issue. But it's much less prevalent now. Related -- scaling algorithms: https://en.wikipedia.org/wiki/Image_scaling#Algorithms

You don't often find browsers disagreeing in 2019 on how to render things, but image colour -- even on the same MacOS (retina) platform -- appear to be one of those things they disagree on still: https://content.screencast.com/users/LouisStAmour/folders/Sn... Safari on the left, Firefox in the middle and Chrome on the right. So you can try it for yourself, the test page is at http://www.ericbrasseur.org/gamma_dalai_lama.html

If this isn't esoteric enough, I know of at least one major broadcast media company (Bell's crave.ca) that doesn't properly encode their HDTV 720p videos in MP4 to flag to the browser they're in Rec 709 colour space, and so the compression artifacts, designed to be invisible in Rec 709 are extremely visible on a wide gamut P3 display.

Of course, these flags were only recently respected on all platforms. Chrome on Windows as of Oct 2016: https://bugs.chromium.org/p/chromium/issues/detail?id=655417... and Firefox (err, actually still not fixed, it depends on the codec...) https://bugzilla.mozilla.org/show_bug.cgi?id=1300170#c87

But colour space is increasingly something you have to pay attention to, as displays improve beyond sRGB -- yet ironically, it's also getting easier to ignore as the latest stepping stone towards Rec 2020, the DCI-P3 colour space, is shared between PC and movie industries. https://en.wikipedia.org/wiki/DCI-P3#2015-2016 (Personally I refuse to buy any display or device that plays back video if it doesn't have a P3 display, ideally OLED)

FYI, you can test for colour gamut device support with CSS to serve wide-gamut colours -- but oddly enough, again, no Firefox support. :( https://developer.mozilla.org/en-US/docs/Web/CSS/@media/colo... FF bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1422237

> back then [in 2009] there wasn't much awareness of this issue

I haven’t noticed too much change in the general awareness of this problem between the 1990s and today.

Experts have always known the right thing to do and people have been talking about these problems as long as there has been image processing, (e.g. Poynton’s Gamma FAQ dates from sometime before 1998 http://poynton.ca/notes/colour_and_gamma/GammaFAQ.html) but it’s new to everyone at some point, and newcomers perennially screw up.

The annoying thing is that there are folks who continue to screw up (like Adobe Photoshop) who should really know better. Backwards compatibility shouldn’t be an excuse to continue with broken implementations for decades.

Oh, the broken implementations don't stop at backwards compatibility -- Lightroom CC, their new Cloud product, last I checked, prefers to send sRGB to Photoshop (!) unlike the traditional Lightroom which fully supports preserving embedded colour profiles when exporting. It's been a couple weeks since I looked into this but the instructions in this post https://lightroomkillertips.com/keeping-your-color-consisten... apply to Classic Lightroom if I recall correctly, and the modern "cloud" Lightroom CC only exports wide gamut correctly when set to export as uncompressed TIFF 16-bit. Everything else tries to export as sRGB automatically, which sucks for print or DCI-P3 workflows.

Also, I'm trying to find details on Apple's P3 profiles in how they differ from standard projection profiles for the movie industry to maintain compatibility with sRGB gamma, but there's not as much online as I expected about it.

In my search for info, I came across https://ninedegreesbelow.com/photography/lcms-make-icc-profi... which seems like a useful resource for Linux/FLOSS folks.

Ah. Found it, a source for the differences between "Display P3" and movie-theatre P3. Here's a link to the relevant section of the video, on colour spaces: https://developer.apple.com/videos/play/wwdc2016-501/?time=2...

And here's a good discussion on why D60 was chosen for film projection while D65 is used by modern computers for sRGB/Rec 709 compatibility: https://www.liftgammagain.com/forum/index.php?threads/dci-p3... and here's a bit on the technical details of ICC profiles and CIE XYZ numeric representations of colours: https://ninedegreesbelow.com/photography/xyz-rgb.html

https://apple.stackexchange.com/questions/252159/what-color-... describes the difference between P3 and what Apple uses.

The same phenomenon also applies to color blending. In a modern context this phenomenon is especially visible in UI elements that blur the background: Up until recently many UIs did not get the blurring right, resulting in greyish dark spots between different colors. I think CSS blurring in most browsers still gets it wrong.

https://www.youtube.com/watch?v=LKnqECcg6Gw (Computer Color is Broken, MinutePhysics)

Well color blending is whole another can of worms; there are no easy answers there. Mixing RGB values even in linear space does not yield always "correct" results. More problematic, I'm not sure if there is even well-defined correct result for generalized color mixing/blending.

Even something as simple as a proper gradient between two colors is surprisingly difficult. I think I finally cracked that one though: https://stackoverflow.com/a/49321304/5987

Color mixing is easy if you're mixing light, just add together the linear intensities. Color mixing of pigments is a different order of complexity, because you have to consider that pigments are both transmissive and reflective and there are other effects like dot gain and spectral lines to consider.

This is over 10 years old, and I just tested 3 browsers - NONE of them got it right!

For image editing at home I use my own software to do resizing, using Lanczos-5 in linear color space.

Browsers do wrong a lot of things. Some time ago I hacked this little test:

https://datenwolf.net/rndrchk/

EDIT: If everything is done correctly, the text in the upper div will remain unreadable.

If browsers were doing gamma correct scaling, the perceived brightness of the checker pattern would not change.

Even more problematic, when zooming the interpolation and blending also don't happen gamma correct. If mere nearest neighbor interpolation were applied, this would remain hidden.

This becomes really annoying on HiDPI screens. For some reason the "px" CSS unit has been "redefined" as an "average-ish" angular distance when viewed on a 96DPI screen at an arm's length or so. And for higher resolutions px is scaled accordingly. Whoever came up with that nonsense created a lot of issues down the road. sigh

EDIT: Also Opera (before it switched to WebKit) did some really funky business at the edges, when upscaling images. Interpolation would always clamp-to-edge instead of respecting the tiling mode.

Regardless if they had included the device relative px in the spec or not CSS doesn't define rendering, it defines layout. I.e. it's intentional CSS doesn't allow you to define things in terms of physical device pixels. Technically you could hack it for real world devices with ton of media queries but that's not their intended usage (hence needing a ton).

If you're intending to render something in the browser (calculating DOM elements, canvas, or anything else) they expect you to handle it in your rendering code (JS/WASM).

The whole intention of my little test hack in the first place was to show, that without being able to define layout with device native unit precision, the end result becomes unpredictable.

The idea was to apply a checkerboard background image to to body and div, but translate the checkerboard by 1px in the div. By their very nature pixel based images are defined in pixels. And the most naive way to display images is by a 1:1 mapping of image pixels to CSS px units. This is how things started out, and only later features like page zooming, responsive scaling, HiDPI and so on came to be. Of course that means that images must be scaled. But this scaling must be well defined, so that the output will only differ in resolution, but not layout or visual outcome after scaling.

And right now browsers fail to do this. Two years ago, when I wrote came up with that test Opera and Android WebView did even fail to properly translate the position of the div background; it looked like if somewhere in the scaling at some point the translation was coerced into the device native units grid.

A checkerboard pattern can be understood as a Haar wavelet; or in terms of spatial frequency space as ΣₖΣₗsin(x-k)/(x-k)·sin(x-l+π)/(x-l+π) When applied to the pixel sampling grid a 2×2 checkerboard pattern is right at the Nyquist limit. Upscaling it with an ideal filtering kernel (sin(x)/x = sinc(x) = Lanczos) yields a single frequency (sinusoid).

Adding two such signals at exactly π phase difference gives perfect destructive interference. Add some phase difference and it becomes constructive. This little detour into signal theory should make it clear, that you have to take great care when scaling and positioning stuff in a layout. Scale corresponds to frequency, position corresponds to phase. And it should be noted, that in a visual signal, phase carries the bulk of information, hence image transformations should preserve the phase, where possible.

That it also shows, that interpolation and blending is broken, too (i.e. doesn't respect gamma) is a secondary outcome.

If by "only later" you mean the original CSS1 spec in '96 then yeah. The web has been relative long enough that anyone who has ever hit "print" didn't have to worry about images being 1/6th the size of the rest of the page.

I don't see how broken interpolation/blending is a secondary outcome. That it works only at "native" resolution is a result of interpolation being broken not the other way around. If it was fixed you wouldn't be looking to use device pixels (again except for something like canvas rendering of a web image editor).

Could you please provide a good reference for implementing Lanczos-5?

Sorry, I don't know of one. You could start with the Wikipedia description at https://en.wikipedia.org/wiki/Lanczos_resampling but that is nowhere near enough detail to do an actual implementation. You might try searching for a Lanczos-2 or Lanczos-3 implementation.

I wrote my first image resizer in 1986, so I've been immersed in this for a long time. I'm considering writing a blog post to go into the finer points, but my blog is kind of empty right now and has been for a long time - don't get your hopes up.

I'm playing with the Dalai Lama picture in Gimp and seeing some interesting results.

If you decompose the image into LAB, then the individual channels exhibit this scaling issue.

But the kicker here is that when you're working with these decomposed channels, they are visualized in the GUI as grayscale.

So, thus, never mind the fact that they represent color; that is irrelevant. Color seems to be a "gray herring" here.

What's fascinating is that the eyes are not fooled; optical blurring and scaling is resilient somehow.

So it really is just about the intensity mapping, underscoring how this is a gamma issue.

The data is contrived so that a linear treatment of the intensities will average out to around zero.

This reminds me of say, the wrongness of mixing logarithmically compressed audio samples just by adding them together.

Ah yes, linear light scaling. Implemented in Kodi and MPV since times immemorial... in Imagemagick too. Like 10 years.

It also improves scaling quality for most resampling algorithms.

What if an image is so contrived that the correctly linearized data averages into a flat, gray signal?

Won't the "correct" scaler succumb to the problem?

The answer is no. Here is why: such a contrived image will actually look gray, and so when it scales to a gray square, that is arguably correct.

We can see this with the Dalai Lama readily. If apply even a modest gamma curve to it to bring up its midtones, it turns more or less gray.

Reply on news.ycombinator.com