Because RGB is not a perceptual color space.
With RGB, a lot of values are wasted encoding colors that are perceptually very similar. Many colors that are “interesting” to humans are encoded in a narrow slice of the RGB cube. Much of the cube is a deep ocean of indistinguishable bluish shades.
Video traditionally uses various luminance+chroma color spaces, which are more perpetually effective than RGB and allow for easy subsampling to save bandwidth on the less important color data.
Each of those smaller squares is made by having red at 0 and blue at 0 at the top left corner, then increasing each along one of the x/y axes until you have red 255 and blue 255 at the bottom right corner. The "green" value is constant for each square, and just increases by one for each subsequent square packed in. At distant/normal viewing you're not going to resolve all those individual colors but more see the averages (to say nothing of the computer itself averaging things to make a zoomed out view or a smaller-dimension rendering).
So you basically have subsquares that start, on average, as "purple" (or let's say magenta). They have no or very little green and mostly are mixtures of red and blue. As you go toward the bottom, green increasingly dominates as what were once the darkest parts of each sub-square become just green and what were the magenta parts tend toward white. Blur your vision a little and the image just looks like a magenta-to-green gradient.
You would be able to construct this image so it would still contain "all the RGB colors" but looked from afar instead like "yellow and blue squares" or "cyan and red squares" instead, just by changing your method for constructing/organizing it. There'll be variation in how well these different variants "blend" and so on just based on how we perceive the different colors, of course.
In terms of that perceptual variation, take for example this: https://allrgb.com/thingy , in particular the little preview in the left. This is what would be the "yellow to blue" kind of ordering, but it doesn't really "blend" as much. The bottom subsquares in particular read as "cyan/magenta/blue" combinations rather than just "blue", and in the top "yellow" isn't really as dominant either. Blur your vision again though and it's a yellow-to-blue gradient.
If you zoom in further, you'd soon realize your screen is nothing but pure red, green and blue subpixels at varying intensities.
If you just think about it as a cube of colour with redness (0-255) on one axis, same with green and blue on the others and then taking 256 slices of that cube it all makes sense intuitively.
I want to do a kind of bubble sort on the pixels and see what it looks like.
To be specific, let's say we take two pixels at random and compare each to their 8 neighbors. If they are closer to the surrounding pixels when swapped, then swap them. Repeat.
I may try this, but I'm lazy, so in case someone else wants to...
It'd also take a ridiculous number of iterations to really notice the noise being added. Perhaps unfeasibly many if you're wanting to go deep and truly rely on random comparisons.
That said, I bet there's some other criterion that would have the desired effect.