Comment by srean | Hacker News Reader

Comment by srean | Hacker News Reader

I right away admit that I am an absolute novice in this space, but I have a few questions. The question I always had is why do we not model it closer to the actual tangible Physics and Biology going on ?

For example, the Physical reality is the different frequencies of light. The Biological reality is that different types of cells on our retina respond with differing intensity to each of those frequencies.

So to my naive mind, a way of modeling color is to have (i) a forward model that map light frequencies to response intensities of the different types of cellular light receptors and (ii) an inverse model that estimates the frequency mix of light from the cellular responses.

That is, have two spaces (i) the light frequencity space (a list of tuple of frequency and intensity/power at that frequency) and the (ii) cellular response space.

Once we have these, we can go from a pigment or excited phosphor to a biological response in a two step process.

From (a) pigment/phosphor (+ frequency mix of illuminating light) to output light frequencities (b) from those frequencities to cellular response.

For all processing, make frequencities the base space to work in (allowing us to change/personalize the forward model).

Yes, the inverse model leads to an ill posed inverse problem, but we are now very knowledgeable about how to solve these.

The frequencies may need to discretized for convenience.

I am obviously a novice and don't know much about modeling color, but this way of modeling seems more grounded in the tangibles. This also gives a way to model how a color-blind person might perceive a picture.

Is the reason that we do not do it this way it's complexity ?

Eager to be illuminated (pun intended).

My understanding is that you're describing the CIE 1931 color space:

https://en.wikipedia.org/wiki/CIE_1931_color_space#Color_vis...

You'll see it is based on the physics and human biology, and is the basis for everything else.

The thing is, this color space isn't very useful for color calculations in the perceptual/subjective sense (e.g. if I just want to change one characteristic, like luminosity, without affecting the chromaticity), so we have transformations to more useful spaces like XYZ, Lab, etc.

There's also the fact human vision is a subjective/psychological phenomenon, so only frequency response is not enough: our vision can map different frequency responses to the same perceptual color (metamerism), our vision adapts color perception based on light source, etc.

Thanks for the pointer.

EDIT Thanks again. This exactly like what I had in mind, in spirit at least.

The problem is that this will only work in one direction. You can calculate the stimulation of the photoreceptors for a certain spectrum, but not the other way around. For example the eye cannot distinguish between purple light consisting of one specific wavelength and purble light mixed by red and blue wavelengths, because both give the same stimulation of the receptors. So there is an infinite number of possible spectra for any given stimulation of the photoreceptors. All we can do is take the stimulation values (X, Y and Z) and convert from there to all kinds of color models and back.

Your approach would make a lot of sense for sensors that are full spectrum analyzers, but the eye isn't one.

You are talking about the inverse problem. https://en.wikipedia.org/wiki/Inverse_problem

Yes because it's not a one to one map we cannot invert the map uniquely. But that's ok, we can maintain a distribution over the possible frequencities consistent with the response. That's how it's done in other areas of mathematics where similar non-bijections arise.

Much thanks for answering though, because I suspect I am asking a very basic question.

You're correct, for what it's worth. I too have always wished that light was modeled based on physics, not on how humans happen to see.

Unfortunately the problem is data acquisition (cameras), and data creation (artists). You need lots of data to figure out e.g. what a certain metal's spectrum is, and it's not nearly as clear-cut as just painting RGB values onto a box in a game engine.

For better or worse, all our tools are set up to work in RGB, regardless of the color space you happen to be using. So your physics-based approach would have the monumental task of redefining how to create a texture in Photoshop, and how to specify a purple light in a game engine.

I think the path toward actual photorealism is to use ML models. They should be able to take ~any game engine's rendered frame as input, and output something closer to what you'd see in real life. And I'm pretty sure it can be done in realtime, especially if you're using a GAN based approach instead of diffusion models.

I see. Makes sense.

No need for ML. This already exists, the keyword to look for is "spectral rendering".

To add to the general thread: the diverse color spaces are there to answer questions that inherently involve how a typical human sees colors, so they _have_ to include biology, that's their whole point. For example:

- I want objects of a specific color (because of branding), how to communicate that to contractors, and how to check it?

- What's a correct processing chain from capturing an image to display/print, that guarantees that the image will look the same on all devices?

Good comment. For people with physics and mathematics skills and intuition, learning or discussing about color or music theory can be a little bit frustrating experience. Ie a lot of it is about traditions and some famous people's ideas and less about how it actually works under the hood. I guess similar things exist in most fields when you dig a little bit deeper. "We always apply this fudge factor to get the correct results". Even something considered hard science.

Of course artists can be very effective with whatever the toolsets they have learned, they sort of can transcend all the obfuscations and actually can express themselves. It's a bit hard to change everything then as the "user's transformations" are then baked in.

It's also true that mathematical models are often simplifications and one has to consider the whole end to end pipeline, where a more "accurate" transformation can yield a worse end result if applied blindly. I'm always reminded of the anecdote of when in the past, at some tv channel, they switched to more accurate weather forecast models. But the audience got worse forecasts, since the resident meteorologist was used to certain errors the old model produced and could compensate those, while the new model had maybe less errors in total but they were different. Happens all the time and why some engineer thinking something is objectively better might not actually be better for the customer...

> For example, the Physical reality is the different frequencies of light. The Biological reality is that different types of cells on our retina respond with differing intensity to each of those frequencies.

Because this isn't true. And there are multiple ways with completely different combinations of intesities of light at different frequencies to have a color that most people will see as the same color. This is because the color receptors of our eye overlap.

> And there are multiple ways with completely different combinations of intesities of light at different frequencies to have a color that most people will see as the same color. This is because the color receptors of our eye overlap.

Of course, if you read back you will notice I am quite aware that the mapping is not a one to one bijection. Hence the need to solve the inverse problem.

One can maintains the inverse image (a set), or maintain a distribution over that, or certain statistics of that distribution. I have a link in my post about what inverse problems are.

We see different colors not because our eyes can tell what frequency a particular spectral color is, but because we have three types of color receptors, each of which is excitied by different broad ranges of overlapping frequencies.

Either my English is very bad or your comprehension is. My understanding is no different from what you say in this comment.

Many of these, like RGB or YUV, are intermediate "simplifications", not what you're necessarily supposed to be working in to generate them. It's like a physics calculation where everyone just decides to randomly stop half way through and use that as a transport medium. RGB exists because that's how the display physically operates, it's how bright those literal subpixels should be. Thus the RGB intermediate transport. YUV (particularly 4:2:2 and 4:2:0) is then essentially a lossy-compression format that's very easy to compress/decompress, and that's why it exists.

If you're doing rendering work, like in a game, those do operate in a more physical domain. That's the so-called "Physically Based Rendering" or PBR that you might see if you hang out in game dev circles.

Most display technologies output RGB light, which means you’re already forced into an output methodology that’s relying on deception purpose-built for human biology and psychology.

Because of that, the layer of your model concerned with the physical reality of light doesn’t seem super useful for basic image input and output.

But of course this layer is incredibly useful for stuff like computer graphics, where it is indeed the case that physically based rendering is widely used for offline CGI rendering (and increasingly for realtime video game rendering).

Short answer

We do model it according to perception, just only when it is useful. Other colour models allow different outputs, printing with coloured ink, displaying on screen by mixing coloured lights, categorising colour, finding how bright it is etc. all need different representations in numbers.

I think this is closer to that than you think. Lab has three intensity vectors. One for each of the photoreceptors in your eye. Anything more than Lab is unnecessarily conplex IMO.

The three Lab values don’t map onto the three different wavelengths captured in the retina.

It’s more like the L is an intensity/brightness factor, and then the a and b values corresponding to the two dimensions of opponent color that neurons capture in the thalamus one step after the eye

In fact, to support your point, it is perhaps questionable from first principles if 3 dimensions not 4 is right. Leaving out tetra chromats and the (partially) color blind, normal human light perception is 1 kind of rod and 3 kinds of cones (i.e. 4 photo receptors, plus some light sensitive ganglia that don't seem to participate in vision, but diurnal regulation).

So, sure, this "4th dimension" (for normals) might be as simple as "candelas" - truly orthogonal, but one does hear an awful lot about "ambient" or "candela contrastive" (a term I just made up) kinds of effects. (EDIT: e.g. in color calibration of projectors in dark rooms vs. living rooms, for example, but I'm sure there are many.) I am just one person, but it feels like candela brightness matters for color perception. So, maybe luminous intensity is not actually exactly orthogonal. Maybe this is all covered in the 1931 CIE documents, though.

Reply on news.ycombinator.com