Beware of Transparent Pixels
adriancourreges.com
adriancourreges.com
> As an Artist: Make it Bleed!
> If you’re in charge of producing the asset, be defensive and don’t trust the programmers or the engine down the line.
If you are an artist working with programmers that can fix the engine, your absolute first choice should be to ask them to fix the blending so they convert your non-premultiplied images into premultiplied images before rendering them!
Do not start bleeding your mattes manually if you have any say in the matter at all, that doesn't solve the whole problem, and it sets you up for future pain. The only right answer is for the programmers to use premultiplied images. What if someone decides to blur your bled transparent image? It will break. (And there are multiple valid reasons this might happen without your input.)
Even if you have no control over the engine, file a bug report. But in that case, go ahead and bleed your transparent images manually & do whatever you have to, to get your work done.
Eric Haines wrote a more technical piece on this problem that elaborates on the other issues besides halo-ing:
http://www.realtimerendering.com/blog/gpus-prefer-premultipl...
I completely agree that premultiplied alpha should be used everywhere. I'd even go a step farther and say that you should use high bit depth linear values too, but that's a topic for another day.
- Slow to convert to linear colour space (requires a pow).. except - GPUs use an 8-bit lookup table to convert input sRGB to output linear value. - This doesn't work for more than 8 bits as the table gets excessively large very quickly.
It's a pity PNG doesn't have flag to mark the image data as being pre-multiplied.
Associated (aka premultiplied) alpha is the _sole_ means to embody both occlusion and emission. Unassociated (aka straight or key) alpha cannot represent these facets.
Consider a candle flame that exists as mostly emission and low to no occlusion. With associated alpha, you can use zero alpha triplets with non-zero emission RGB to represent this real-world scenario. With unassociated alpha? Impossible.
I don't think associated alpha helps much here. You can special case pixels that are doing pure emission, but when a pixel is doing both you need the lighting to affect the color of the occlusion but not affect the color of the emission.
But that means you need to dedicate specific objects to being purely emissive, to avoid blurring at the boundaries.
And once you've separated the objects, you don't really need to have purely-emissive textures and non-emissive textures in the same file, with the same exact format. You might as well store emissive textures as RGB and save on memory.
Using a different format can even benefit you. You're less likely to accidentally blend emissive and non-emissive pixels, and you're less likely to accidentally apply lighting calculations to emissions.
No hacking needed.
Let's have blue-tinted pane of glass, (0, 0, .5, .5). And a red tinted pane, (.5, 0, 0, .5).
Then a blue glow, (0, 0, .5, 0). And a red glow, (.5, 0, 0, 0).
If you have your blue glass glow red, and your red glass glow blue, both combinations come out as (.5, 0, .5, .5).
Under 99% of lighting conditions, it will look wrong. If you put it in darkness, it will look overwhelmingly wrong.
Even if occlusion and emission are the same color it doesn't work. A dark blue object that glows brightly, and a bright blue object that glows dimly, both will have the same RGBA.
Anyways, read the links I provided above.
Objects that block a certain amount of light, and then emit a certain amount of light: You can only make that simplification if the entire world is evenly lit by white light.
Blur the entire matchstick into one pixel. Under white light it has to emit brown plus yellow. In a dark room it has to emit just yellow.
It's a clever technique but I can't figure out any way it's not fundamentally incompatible with having lighting. If one pixel has to both be lit and emit extra light, you need two RGB values.
It is complimentary to lighting.
The math works because of the differing forms of the alpha over formula. FG.RGB + ((1.0 - FG.Alpha) * BG.RGB), resulting in a pure add at the extreme case of alpha being zero, or ratios of addition when non-zero.
Within the limitations of the RGB model, it works extremely well.
The folks cited are extremely adept imaging people, covering years of experience and several Academy Achievement Awards.
You don't know what color will be emitted unless you know what light is hitting the surface.
A dark glowing surface and a bright non-glowing surface have the same emissions under white light, but different emissions under other kinds of light. The method you're talking about requires the emissions be precalculated, which means you can't apply lighting at runtime.
The main reason why CG studios can't use PNG is because both renderers and compositors always output pre-mult images. Yes you can choose to un-pre-mult them after rendering, but the native result of a blending operation is always a pre-mult color, regardless of the input sources. Unpremultiplied blending still results in premultiplied colors.
Since everyone knows that un-premultiplying (dividing) is to be avoided at all costs, it means you can't render or comp something and then save the file in PNG.
> Associated (aka premultiplied) alpha is the _sole_ means to embody both occlusion and emission.
I would suggest avoiding thinking of image colors as emissive. That's a material property, and using RGBA to encode material properties is only something you'd do if you were stuck in a weird fixed-function pipeline with no choice, or if you were really really low on disk space. Otherwise, emission colors go in their own separate emission channel that doesn't have an alpha value.
Feel free to reference the original Porter Duff paper regarding "luminescent pixels" or Alvy Smith's opinion on the matter as relayed by Zap Andersson in the legendary Adobe thread.
https://forums.adobe.com/thread/369637
Here is a sample of the candle referenced by Zap:
https://i.imgur.com/bnXyAl1_d.jpg?maxwidth=640&shape=thumb&f...
Remember that a ray tracing engine uses associated alpha as that is the sole format it can generate. Only associated alpha models emission and occlusion.
Further reading:
http://lists.openimageio.org/pipermail/oiio-dev-openimageio....
https://groups.google.com/forum/m/#!topic/ocio-dev/ZehKhUFqh...
Yes you're right; I wasn't arguing with you about that. I always feel like calling it additive rather than emissive. But just because you can represent additive colors in premult images doesn't mean you should, and I'd speculate wildly that it occurs less often in production than halo problems. Someone who writes lens flare and rainbow shaders is going to scold me for saying that though...
In the context of the OP's article, and of artists who paint images with transparency in them, worrying about emissive colors isn't really an issue. Artists very rarely paint pre-mult images, they can't work with premult images, generally speaking. No doubt a few people who know what they're doing do it, but I can't personally say I've ever seen an artist painted premult image with emissive colors, nor do I recall ever seeing a software rendered layer with emissive colors either. Is this common now? I've been out of film & games for a few years now.
I'm already familiar with everything you referenced; and I can vouch that it's all very good stuff so thanks for sharing, especially the Adobe thread. I hope others here benefit. It's amusing that an entire industry knows who Chris Cox is because of this thread, right? :P
If the software converts to pre-multiplied, then there'd be no halo problem and no bleeding necessary, right?
You're right that bleeding won't hurt assets if you're comping, but it will hurt assets if you're not converting to premult and then do texture filtering or mipmapping or blurring.
My concern is with using bleeding is the article's suggestion to use bleeding as a first resort, rather than a last resort (as an artist). It's a hack that totally works in a lot of cases, but it's still a hack. I've watched artists in film and games use random combinations of bleeding, (un)pre-multiply, gamma, and other stuff whenever something goes wrong with matting, and often it's not the right solution. A lot of people are scared of understanding premultiplied alpha - and the technical name isn't doing anyone any favors - and instead of figuring out the right solution they try every combination of hacks until it works. General misunderstanding and superstition about premultiplied alpha is the most common reason I've seen for people using un-premultiply nodes in production.
[0]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2741
Wish the article was more clear as to why this happens. Let me elucidate: this happens because, per the PNG standard[0], 0-alpha pixels have their color technically undefined. This means that image editors can use these values (e.g. XX XX XX 00) for whatever -- generally some way of optimizing, or, more often than not, just garbage. There are ways to get around this by using an actual alpha channel in Photoshop[1], or by using certain flags in imagemagick[2].
[0] https://www.w3.org/TR/PNG/
[1] https://feedback.photoshop.com/photoshop_family/topics/png-t...
[2] http://www.imagemagick.org/discourse-server/viewtopic.php?t=...
If you follow the recommendation at the end to use premultiplied alpha for all computations, this becomes a moot point.
In an associated alpha image (aka premultiplied), zero alpha with non-zero RGB represents emission with no occlusion.
The issue with the Limbo logo was not that the source image was incorrect. The image was fine. The blending was incorrect because the PS3 XMB has a bug. Not using premultiplied alpha when you are doing texture filtering is a bug.
The correct solution is to pay close attention to all of the factors... and to be ESPECIALLY aware of pixel scaling. Provide your RGBA textures at the 1:1 pixel scale they will be rendered (or higher!) if at all possible.
Disclaimer: not a graphics programmer just a hobbyist seeking clarification. :)
For image editing, the problem is that you might send the same pixel through many different operations, and if you flatten something and further manipulate, then the rounding error will continue to accumulate. So losing granularity early in the process due to alpha can result in accumulated rounding error.
In 3d graphics, the texture typically goes through a very predictable and short number of transformations, and they will rarely need to "stretch" the color range of the transparent pixel. When you alpha blend with pre-multiplied alpha, you're literally just adding the whole color value. And then you're slapping a whole lot of other colors onto it in the lighting/shadow/other passes, so the subtle nuances of that mostly transparent window get lost in the bustle.
In practical applications the reduced color gamut isn't a problem, your ability to discriminate the colors goes down as the transparency goes up. It would only be a problem if you were trying to convert the pixels to something less transparent than they started.
Yeah, that's not the definition of "1:1" that I use. The definition of 1:1 that I use has texels lining up exactly with pixels.
That doesn't matter unless you color-scale the image (like multiply by 2 to make it brighter) before displaying it. Otherwise, the depth is at the correct resolution for display.
And premultiplied alpha should be used for final display, not just for the halo-ing reasons demonstrated here, but for lots of reasons.
Artists should generally be working in un-premultiplied alpha though, and the premultiplication is something that should happen right before an artist image is used. Artists shouldn't work in premultiplied images (and they generally don't) because of the color depth issue, and because it's crazy to paint premultiplied transparency manually.
It does if you stack several image layers. Let's say I have a particular color tone. Then I use that as background color. And I also stack 10 layers of the same color with alpha 0.05 on top of that. If you use premultiplied colors then this will actually result in a different color.
Due to rounding those colors often tend to be more greyish too. So if draw some vector graphics and have multiple basic shapes with semi-transparent edges (aliasing!) stacked on top of each other you can get some ugly fringes.
Even if it did happen, the error is bounded - with 10 layers of the same color, the maximum error in any channel is 5, and the average error is 2.5. It's pretty hard to say it would be wildly and noticeably different to most people even with 8 bit color channels, but I certainly have met some film directors and CG supervisors who were very color sensitive. It would be literally invisible in anything higher than 8 bits.
I'm curious -- why do you say the rounded colors would tend toward gray? Rounding error can happen in both directions, so I would expect rounding errors to cause a uniformly distributed error -- some colors would get slightly more saturated, some less, some of them would shift hue, and some would be unaffected.
You lost me on the edges part & aliasing. Rounding errors will not be visible as fringes, so if you're seeing fringes, you have some other problem... perhaps failure to pre-multiply! ;) Tell me more about stacking shapes and getting fringes. Is this stacking 10 of the same shape in the same place, or at the edge crossings of different shapes? What kind of fringes are you seeing?
Anyway, I've never witnessed a case in 20 years of film & game production where rounding errors caused a visible, detectable problem, but I'd love to know if there real cases where it's an issue!
In a toy project where I tried to automatically generate SVG shapes and rendering them to a html canvas (which uses 8bit per channel premultiplied alpha). The assembled shapes occasionally overlapped and it did lead to visible color inconsistencies.
It's also a problem when working with PNGs. When you put your PNG pixels into a premultiplied space and pull them out again you actually lost information to rounding, which negates the benefits of a lossless format.
> You lost me on the edges part & aliasing. Rounding errors will not be visible as fringes, so if you're seeing fringes, you have some other problem... perhaps failure to pre-multiply! ;)
Well, if you got a solid shape then there's no alpha. But the aliasing at the edges introduces partially transparent pixels. If you then pull out the pixel data and apply it to a different canvas you get mismatched colors.
I only spent a few hours on it, so I don't recall all the details, but my conclusion was that it is inadequate for image manipulation since it is lossy and lacks precision.
Although if you want to keep 32-bit textures you also shouldn't be doing linear premultiplication like the article suggests, you should be using sRGB instead.
One clarification, though: With premultiplied colors, something like (1,1,1,0) is either illegal or a light source. It's not a valid normal color.
Also: "The original color can still be retrieved easily: dividing by alpha will reverse the transformation."
C'mon, you can't say that and then make an example with alpha=0. Do you want me to divide by zero? The ability to store values in completely transparent pixels is lost.
There're no channel formats or clever equations I'm aware of that avoids the redundancy part. But your question totally reminds me of Greg Ward's RGBE format, which is a high-dynamic range format stored in 8 bits per channel, with an extra 8 bit exponent channel. http://www.graphics.cornell.edu/~bjw/rgbe.html
RGBE isn't doing exactly what you're asking about, but it's similar in a way. Instead of storing 16 bits per channel separately for each channel, what you really get instead is 16 bits (sort-of) for the brightest channel, and the other 2 channels in 8 bits each - discarding the extra bits. You can't see them because the bright channel will prevent you from seeing anything super dark in another channel, so you can discard the extra color resolution in the darker channels.
If you count compression, then pre-multiplying would help the situation. Anytime the alpha value gets low or goes to 0, the color channels do too, so run length encoding or DCT or whatever else will collapse large transparent areas into all zeroes.
I don't understand this. When I make transparency I don't use any color? I use the Eraser tool or Ctrl-X, not a color with 0 opacity.
Here is some information for Photoshop and a plugin you can use:
https://graphicdesign.stackexchange.com/questions/63783/how-...
If you inspect the image in a full-featured editor like Photoshop or GIMP, you can inspect the channels individually or remove the transparency entirely to see this fact.
I think with most GUI programs, the eraser and cut tool will leave color as what it was before it became transparent.
EDIT: saurik has a good point that I forgot about -- many editors may actually throw the colors away when you export unless you ask them not to.
As the author mentions, certain resampling algorithms might be naive or plain ignorant about how to resample images with potentially transparent pixels. Should transparent pixels not count to the final pixel? Should all pixels be averaged? Should the output pixel be the median or mode value? If the image is being resampled to 1/3 its size the resampling can be very cheap if only the middle of the 3x3 pixel cluster is selected as the output value.
Say you have two adjacent pixels using floating point RGBA values of (0,0,0,0) and (1,1,1,1), and you apply it to a 3-d shape. Because of the rasterization algorithm, you will be sampling weighted averages of the two pixels, either because you're scaling up and need to interpolate, or because you're scaling down and need to average.
The average of (0,0,0,0) (fully transparent) and (1,1,1,1) (opaque white) is (0.5,0.5,0.5,0.5), a half transparent gray. But you'd intuitively expect (1,1,1,0.5), half transparent white. This is the essence of the problem. The fix is to make sure that your transparent pixel was (1,1,1,0) and not (0,0,0,0).
Did you mean low opacity?
If so, that's not quite right. (0.1,0.1,0.1,0.1) premultiplied is the same color as (1,1,1,0.1) "normal." They're both white and low opacity, just in different representations. You don't actually lose much granularity because the graphics card has to multiply the color channels by the opacity value sooner or later.
Separately, your formula doesn't work for interpolation. It works for averaging, but in order to do texture sampling, you need interpolation, so your formula can't actually be used unless you can adjust it to deal with interpolation gracefully.
I don't quite get your point on interpolation, but I'll look up when I have the chance
An alpha mask (on a layer with no alpha channel) is essentially a different way of viewing & editing the same data, and there you probably have no logical trouble with using a paintbrush tool.
https://github.com/w3c/svgwg/issues/303
It affects gradients, animations and imported+scaled raster images. Maybe other stuff too, I don't know.