A Pixel Is Not A Little Square (1995) [pdf]

Color and luminance is a separate and orthogonal issue from filtering. I also know that people get away with compositing without converting to linear space, but I'm skeptical that any benefits they see aren't just a matter of getting the color curve they want for free, as opposed to doing a similar correction after something has been composited correctly.

Gamma curves do not affect black and white themselves, only intermediate grays. It’s true that grays are usually used to draw antialiased black and white lines, but we can also think about “ideal” (axis-aligned, pixel-centered, non-antialiased) black and white lines. By framing this as a question about gamma, you’ve implicitly assumed that “ideal” black lines on white would have equal perceived thickness to “ideal” white lines on black.

This is not the case, as typographers have known for decades. The right way to draw perceptually equal-thickness black-on-white and white-on-black lines is to vary the line width (perhaps even by more than a full pixel if the lines are thick enough!). Gamma only comes in afterwards, to help us reproduce the varied widths accurately, and the accurate way to do that is in a linear color space.

The perception of equal line thickness may depend on orientation of the lines (https://res.mdpi.com/vision/vision-03-00001/article_deploy/v...), distance from the fovea, distance from where your attention lies (a can of worms even deeper. I don’t think there’s agreement on whether one can attend to more than one visual location at a time, or what ‘attention’ even is), direction of those distances (vertical will almost certainly be different from horizontal, sight could be better or worse in the nasal direction vs the temporal one), light/dark adaptation of the eyes, whether subjects are color-blind, etc.

I fear that, to qet something that’s better than Weber’s law (https://en.wikipedia.org/wiki/Weber–Fechner_law), you need a model so complex that it doesn’t make sense to start using it.

I break every mathematical object down into three things:

1. The intuition. Why do we have this concept to begin with? What underlying idea are we trying to capture?

2. The definition. These are the axioms.

3. The implementation. This includes every way to communicate the idea, from natural language words to notation to source code.

Without the intuition, you have nothing but a symbol game. It's hollow. Something with rules and notation but no deeper intuition is, arguably, chess.

Without the definitions, you can't think rigorously about your ideas and you don't know if they lead to internal contradiction. You can dump the axioms without losing the intuition; we did this with set theory at the turn of the previous century, when Russell proved that the previous axioms were inconsistent. We saved set theory without having to abandon the notion, the intuition, of sets entirely.

Without implementation, it's just thought, and you can't communicate with anyone. Moreover, without some intuition, the implementation is meaningless, because you have no cognitive frame to use to interpret it.

So the tuple of numbers is one implementation of a vector. It allows you to communicate some aspects of a vector, but without the underlying idea of what a vector means, what concept we're trying to get across, it's just a list of numbers. They might as well be box scores or something.

This is not accurate. Computers generally represent raster images as arrays of numbers (where each entry in the array is called a “pixel”). There are no literal little squares involved. Some code (much of it mediocre) conceives of those arrays of numbers as representing little squares. Other code does not.

> vectors-as-numbers is much more useful when calculating with them

This is super myopic / parochial.

Mathematicians think of “vectors” as elements of an abstract vector space (i.e. anything with well-defined concepts of scalar multiplication and vector addition over some field). This is useful to them because there are many powerful theorems which work in general for any arbitrary vector space, or sometimes for any vector space over the complex number field, or sometimes for any finite-dimensional vector space, or ....

Physicists think of vectors as directed magnitudes, generally some kind of measurable physical quantity in Euclidean 3-space (or Minkowski space). This is useful because many kinds of combinations and relations of directed magnitudes can be computed can be made without reference to any specific coordinate system.

One possible representation of physicists’ vectors (or certain types of mathematicians’ vectors) is an array of numbers.

But an array of numbers by itself is a completely different type of object than a vector. There are no specific well-defined operations on a generic array of numbers; or rather, depending on what it represents there are a wide variety of operations that might be meaningful or reasonable.

There are many kinds of “calculations” which are completely abstract where thinking of vectors as arrays of numbers is unbelievably obscurantist and counterproductive. Proofs and derivations involving coordinates are almost always extremely cumbersome.

There are even many types of concrete calculations on vectors-represented-as-arrays-of-numbers where the most effective algorithm is to first convert to a different representation.

Mathematicians like to think of just about everything as vectors, so that sentence seems a bit off.

Anyway, yeah, starting from abstract definitions is a great way to make a student's life hell. After all, there's a reason universities teach linear algebra twice.

There’s certainly a sweet spot, where too much theoretical background takes away from learning graphics, and too little could leave students unprepared, or worse, uninterested.

It’s certainly good & fun to go through at least a little sampling theory.

You might also be interested to know the guy who wrote this paper co-founded Pixar. His graphics advice is worth careful consideration.

> It turns out that thinking of them as little boxes arranged in rectangular grids is very useful.

Do keep in mind that the shape and the arrangement are two separate things. Alvy’s paper is talking about the sample shape, but not talking about the arrangement. The grid arrangement is useful, and Alvy would agree.

> Because that is how computers deal with them. Not as point samples.

I’d be cautious drawing that line. I realize you were talking in part about grid arrangements. But computers treat samples however we teach them to. It’s uncommon and unlikely you’re writing a lot of code that truly handles pixels as finite square geometry rather than a point sample. Plus, if you’re teaching things like magnification filtering using bilinear or bicubic, then you’re already treating pixels (or perhaps texels) as point samples.

Here’s what you get when you treat pixels as squares: https://news.ycombinator.com/item?id=17845460

One cool trick is that you can sprinkle in more efficient white pixels to allow the display to reduce its power consumption.

[1] https://en.wikipedia.org/wiki/PenTile_matrix_family

Except for a philosophical debate about continuity, isn't that true?

https://youtu.be/cIQ9IXSUzuM <-- Youtube backup in case it doesn't work (YMMV)

I wish I could force any audiophiles to watch it before they waste their money on snake oil. I think a similar point can be made about analogue synthesis, e.g. You pay Moog £1500 for what is ultimately a fancy box for a relatively simple analogue circuit.

There are two ways this is not true for pixels. First, even for "retina" displays the human visual system can make out spatial frequencies beyond the Nyquist limit of the display (this will vary by viewing distance, so is more of an issue for young people who can get close to their displays). Second, even assuming perfect gamma, the display must clip at black and white because of physical device limitations. Thus, especially for text rendering, only a reconstruction filter with nonnegative support is generally useful. Such a reconstruction filter would be an extremely poor choice for audio.

It is true that many of the underlying signal processing principles are the same, and I encourage people to learn and understand those :)

For example, you can also have “pixel art”, drawings made up of little squares, which only loosely have to do with pixels as samples for a raster image or physical camera detector elements or physical display elements. https://en.wikipedia.org/wiki/Pixel_art