Premultiplied alpha vs. not-premultiplied alpha blending and compositing
iquilezles.org
iquilezles.org
(Not the worst YT to reference anyway, in my view)
* 50% curiosity about the coincidence
* 50% readers that liked this article may also like ...
But yes, you must do your blending in linear values, otherwise what you get is nonsense.
What do you mean, and why do you think this? The definition of the term “nonlinear color space” is one that has a nonlinear transfer function. What does it mean to say that all color spaces are defined in terms of linear intensity, exactly? (And in that case, what would a non-linear value be?)
The point is that a "color space" is the definition of the physical characteristics of the individual channels. This is always linear. After that, many color spaces do define a transfer function for storing and transmitting the data, usually meant to be used for cases where not much precision is available, e.g. when you only have 8 bits per channel. So to say that you have to "switch to a linear color space" is incorrect, as you are not changing the color space, you are just transferring (applying the inverse transfer function) the values from their non-linear representation to linear.
You can nitpick and say that the "sRGB colourspace" is linear but it's still true that sRGB is nonlinear.
A value is only “linear” with respect to something else. Not all values are linear. In the case of color, the words ‘linear’ and ‘non-linear’ are used with respect to physical measurements of light intensity such as watts per steradian or radiance.
sRGB values are not linear, because those values are not a linear transform away from physical measurements, it takes a non-linear function to map from physical measurement into sRGB.
The reason sRGB is non-linear is because when you add or multiply two sRGB colors to simulate physics like emission or reflection, you get the wrong color compared to what happens in the real world. To get the right color, you must transfer your sRGB inputs to linear space, combine them there, and then transfer the result back to sRGB space.
> many color spaces do define a transfer function for storing and transmitting the data, usually meant to be used for cases where not much precision is available
This is mostly wrong. The transfer functions define the color space, and they’re almost always defined for the purpose of emulating either display devices with non-linear response or human perception (which has non-linear response). sRGB’s non-linearity is emulating the non-linear gamma of your monitor https://en.wikipedia.org/wiki/SRGB#Transfer_function_(%22gam...
Data space savings is almost entirely a byproduct of more closely matching human perception, and data savings is more or less only relevant to 8-bit color data or less (because 8 bits is not enough to represent linear color without getting visible color banding, but is almost enough when using a non-linear color space like sRGB, while 9 or 10 bits actually is enough to keep things in linear without banding). You still have to switch to linear color spaces to blend colors, even when using 16 or 32 bits per channel, so the compression side-benefits have very little to do with whether you need to use your transfer functions to switch between color spaces.
> So to say that you have to “switch to a linear color space” is incorrect, as you are not changing the color space, you are just transferring (applying the inverse transfer function) the values from their non-linear representation to linear.
This is entirely wrong. Using a transfer function or inverse is always a change from one color space to another, by definition. That’s exactly what a transfer function does, and it’s the only thing a transfer function does: changes (transfers) values from one color space to another. There is no such thing as a ‘linear representation’ of sRGB, there is only sRGB space and linear space. You aren’t making sense with the “non-linear representation” part, you just said all color space values are linear (even though that’s not true).
When you linearize sRGB you create a new not-sRGB-anymore color space that happens to have a compatible white point and RGB primaries.
Do you know if this new color space has an official name?
I work with it basically everyday via the three.js renderer internals and mostly see it refered to as simply "linear", but if more specificity is needed usually I see "linear sRGB". I think that's common even if it's not correct.
However as HDR color spaces take over from sRGB I expect it'll become more important to have a correct name for this color space. I have searched for one previously but haven't been able to come up with anything better than "linear sRGB".
You might not ever need the physical scale factor, it doesn’t generally matter. You probably didn’t set your light or color intensities in three.js using physical units in the first place, and even if you did you might be simply scaling the values until things look right. We usually scale the inputs and outputs manually anyway, so the absolute scale of the intermediate linear color space in CG is frequently irrelevant.
HDR is mostly referring to whether your values are conceptually [0..1] or [0..infinity]. Sometimes people are also referring to how many bits get used per color channel. (But note that the R is HDR is referring to ‘range’ and not to ‘resolution’.) 8 bit color values that represent [0..1] are unambiguously LDR, while 32 bits per channel color values that represent [0..inf] are unambiguously HDR. If you use 16 bit colors and your range is [0..3] to allow for headroom and glare effects, it’s not really considered HDR, but it’s not exactly LDR either. It mostly only makes sense to work with HDR values in a linear color space, but it’s not strictly or by definition necessary.
“Linear sRGB” gets used sometimes, and what it means is that a color was converted from sRGB into linear. So the space is actually just linear. It’s useful to know if a color came from sRGB sometimes, because there’s less worry about going back into sRGB if you know it came from sRGB originally, and it helps you understand something about the scale and brightness of your linear values. If you have linear values and don’t know where they came from, you have no point of reference, and no idea if a value of 0.1 means invisible black, blinding white, or just middle gray. But unless I scale the linear colors, I do know a linear sRGB value of 0.1 is a dark gray that’s visible, because I know that the range of color is approximately [0..1] because it came from sRGB.
Nod... unless you're using a color space, like Jzazbz, with an absolute (not relative) transfer function. Application context then motivates the nit pick. So a Jzazbz HDR of [0..10k], might get a daylight 1k nits picked for diffuse (non-highlight) white. Thus I appreciate it when color libraries provide a distinct Absolute XYZ D65 data type.
I want to elaborate just slightly on what I meant. In CG, like in real photography, the photographer almost always sets some combination of their exposure, aperture size, and/or white balance values manually, regardless of how well they’re controlling inputs and color spaces. As a simple example, I can always dim the lights by a factor of 2 power, and also either expose for twice as long, or open my shutter by one f-stop. When I dim lights by half and expose by double, then nothing about the image changes. Even if you know the physical luminosity of your lights, the overall process (usually) still cancels scale factors and at the end reduces to an arbitrary scaling that essentially sets the output white and output black to the brightest and darkest colors I want to show, respectively.
With real film, there are additional arbitrary scaling values like the film development process, and lighting conditions when viewing. With digital, the display’s scaling is frequently unknown, and thus arbitrary; you might use 1k nits for your daylight diffuse, but that’s not what your monitor actually shows you.
There are legit use-cases for wanting absolute physical values in the intermediate stages, and using a “linear” color space for blending is, in a way, emulating exactly that desire but still with an arbitrary scale because the input brightness and/or output brightness are being tuned by hand.
I needed a color space for teaching, so I'm using it to kludge wide-gamut hue linearity onto CAM16UCS. I wish I knew of something better to do.
Backstory is, I've long struggled to find community of interested and discussion, around the idea of transformatively improving science education content by collaboratively applying vastly greater-than-usual domain expertise. I thought to troll with a worked example, and chose "color". Even though "I'm building hard thing solo, because I really don't want to build such things solo?!?"... sigh. Color is pervasively taught in early primary, but with such poor content and outcomes, that foundational confusion persists even among first-tier physical-sciences graduate students. So: might color be taught more clearly by emphasizing spectra, in early primary? With web interactives? It seemed a question to explore.
So I needed a perceptual color space optimized for learning color. A rather different objective than usually drives color space design. Computability doesn't matter, as small lookup tables suffice. But noticeable features from eyeballing the space should be "yes, that's color perception", not "oh, no, that's an artifact of the model - try to ignore that aspect". The usual art/science education "one part careful, nine+ parts misleading bogosity" just doesn't end well. CAM16UCS is generally good at avoiding "that... doesn't look plausible". Better than other UCSs I skimmed. But... hue gets curvy near its edges, especially badly in blue. Jzazbz regrettably fails the "space looks plausible" test, but is wide-gamut, and has nice hue linearity. So I'm using it to kludge greater linearity onto CAM16UCS, between sRGB gamut and visible boundary. Which aids understandably coupling space to spectra.
Getting from daylight to banana to spectra and apparent spectra and color space, provides a lot of opportunity to get things wrong. And from interactive draw-a-spectra. Swapping in Jzazbz for end-to-end HDR physical units provides the warm fuzzy of sanity check.
Thus three.js intensities in nits, and an opportunity for Sunday punning.
It's just a silly niche use case, rather than advanced use, but modulo burn out, I'll be back at it tomorrow. Also... there's an order-of-magnitude luminance "why don't you see a sky full of stars" interactive somewhere on my infinite todo list, so getting perceptual luminace right had additional motivation. Though that will have to handle meso and scotopic perception... sigh. Appreciated your comments.
I had a few good teachers along the way who taught me a ton about color and computer graphics. Peter Shirley is probably the top of my personal list, and he’s also very interested in improving science education, so maybe worth checking out his CG books. In his rendering class we bought a MacBeth color checker and had to take a photograph of it set inside a scene that we physically built out of whatever materials and light sources we wanted, and then we had to write our own renderers to match the photo. You really learn a lot about color trying to do that! :)
Pete and I have since designed a simple opponent-process color space aimed at ease of use that is perceptually uniform ‘enough’, but we haven’t published it yet. That could be a fun way to teach though: have students design their own color space! There also might be some interesting work going on in the area of digital color picker interfaces that could be useful for teaching. I’m forgetting who does this stuff, so can’t cough up any links at the moment, but I’ve definitely seen a couple of neat presentations recently that combine the science of perception and color spaces along with good design sensibilities to make interfaces that are demonstrably better for artists than the crappy RGB & HSV & other pickers we usually see today.
Oh yeah. I was (hmm, am) hoping to dig out some limited cluster of concepts that gells as "oh, that's fun, and I'd very not have thought you could teach that, or that way". Eg xkcd color name regions in a voxelized color space as "learn color names" K-1 worksheet. Trying for a "light"(spectra) vs "color"(human perception) distinction, limiting scope on perception to early primary color topics.
Science education rarely does deep, integrated, or usable, so it's rather an open question what might be possible. And without a vision of that, there's little incentive for funding or exploring other than incremental improvement.
Even aside from creation pragmatics. "A big onboarding welcome to all our new science textbook author staff! Worry not that you are fresh liberal arts graduates, with no science background at all, for we've A Scientist on call!"(paraphrased)
> a simple opponent-process color space
Ah, neat. That could be fun.
Given my focus on spectra, I was also tempted by IGPGTG's "build on Gaussian spectra", but pruned.
> have students design their own color space!
Oh, there's an intriguing idea. I'd though as far as "if I can't find a color space without misleading large artifacts, I'll have to offer a diverse swappable several, if students are to have any hope of distinguishing data from noise". But a direct manipulation "create your own color space"... hmmm. Perhaps evaluated with visual metrics? Like gradients for linearity. Or mixing for Euclidean-ness and hue order - reorder a hue circle, and get weirdness in the in-betweens? Neat approach - not "this is just how it is" but "hands on, you'll find this a sweet spot". Hmm, if say, "Pink is the most important color! So its variants should be the focus of my space!"... what might that look like? A "pink-set" of color names? Eg, not "green" but "anti-magenta"? Not "white" but "palest pink"? :) I so miss pre-covid brainstorming like this at MIT. :/
> a couple of neat presentations recently that combine the science of perception and color spaces along with good design sensibilities to make interfaces that are demonstrably better for artists
Very neat. If a link surfaces, I'd be interested. There's some work on simulating pigment spectra and interaction for "physically realistic" painting, and I wondered at their UI. And if a MVP web app with such, might serve as hands-on antidote to the "subtractive rules are peers to additive" misconception.
> take a photograph of it set inside a scene that we physically built
I wondered how controlled lighting (filter gell on box, discrete LEDs, and tablets getting the narrow spectra of quantum-dot displays) might be leveraged. Eg, "sketch what you think this page/thing/scene will look like if lit like X, then try it and describe it, and take a picture"?
And sketched, sort of a point renderer - a block ui for playing with spectra. Eg, a Sun block, realistically pictured with red-tinted rim on white, with a spectra and white color", passed though atmosphere block set at angle X, with this transmittance/reflectance/florescence strip, yielding this spectra and color, lighting this point on a multispectral image of fruit, with/yielding ditto. Seen by this camera to get rgb, shown by this display to get this spectra. Banana vs banana pixel. Filters, bounced sources, etc. And then perhaps challenges of "assemble blocks to get a spectra like this". Or to light this multispectral MacBeth or scene to match this snapshot. Or bantering, "your freehand or musically-keyboarded spectra looks like this strawberry icecream with blue M&Ms". A hands-on way to address the many misconceptions around lighting and color.
And then there's video shaders, filtering or fragmenting by color space and names, for interacting with color in the environment. And... the minor challenge of finding a coherent MV"P" in all of this. :)
The choice of the inverse gamma curve you're using is a choice with no single answer depending on how you intend to map that linear light onto...whatever it is you're doing. The obvious one is to use the nominal gamma of 2.2. That may or may not make sense depending on your display technology and what you're doing.
In this older terminology, pre-multiplied alpha is called associated alpha. He goes through the math, but a key takeaway is: "... downsampling and, in fact, all filtering operations should operate on arrays of associated pixel colors ..."
I really enjoy Blinn's articles (and iq's :).
I wish I could find the article I read that argued for this. It had a lovely demonstration of how you could use this to combine fire and smoke effects into a single texture (smoke occludes but fire effects tend not to).
http://www.realtimerendering.com/Principles_of_Digital_Image...
Also these days there may be better and more focused resources online if you know where to look.
http://poynton.ca/notes/colour_and_gamma/ColorFAQ.html
https://www.handprint.com/HP/WCL/color2.html (https://www.handprint.com/CE/book.html)
https://www.realtimerendering.com/
http://paulbourke.net/miscellaneous/colourspace/ (Old but still useful)
The original compositing paper (which didn’t address issues of perception or non-linearity): https://keithp.com/~keithp/porterduff/p253-porter.pdf
You may also be interested in "ACM: Transactions on Graphics", the Association of Computing Machinery's publication of computer graphics research papers. I suggest going to a University technical/research library, where you should be able to access the collection of issues from the 80's, where the original scan line, ray tracing, CSG, and pretty much every single advanced graphics technique (minus the deep learning) used today is documented by the original innovators.
At that same University research library they might have the collected set of course text books (mimeographs and photocopies) used for the 3-day long courses taught at SIGGRAPH every year.
These items are invaluable, and I reference them multiple times a year.
There is some debate over whether the alpha should be stored linearly or not in low precision (8 bit) formats. I think the standard practice is linear, but there’s good arguments it should be gamma encoded.
And, from there it’s important to point out that while pre-mul alpha is great for the math, having a color recorded in an 8-bit per channel image pre-multiplied doesn’t leave many possible values in the low range. So, you can get a lot of banding. Dithering becomes important. You should probably store the image non-premultiplied and do the premul as part of the math at runtime. Unfortunately, almost all image editors and encoders make it impossible to control the rgb values of low or zero alpha pixels. They all assume those colors conceptually “don’t exist” and don’t matter. But, they matter here because the filtered transition between solid and transparent pixels is the whole issue we are struggling with here.
I recommend reading the technical memo "Alpha and the History of Digital Compositing" by Alvy Ray Smith [1] to get a better intuition on the matter.
"PS3 does sRGB conversion before alpha-blending, so the blending is done in gamma space, which is not quite right."
original image -> convert to linear color space -> multiply alpha -> blur -> composite -> possibly convert back to sRGB if needed
The wrong way I meant was
original image -> multiply alpha -> gamma-correct-blur -> composite
where gamma-correct-blur = convert to linear color space -> blur filter -> convert to sRGB