The color vs black and white split also has biological analogies.
With that said, I'd hesitate to dismiss the GP comment. Separate from the color vs grayscale split, why do we observe low-frequency preferring to group with color? It seems very plausible to me that if there's a systematic artifact from how the data neural networks are trained on was compressed, that could play a role. Either way, it makes the argument that this is emerging from purely natural data and the network less clean. (One caveat is that these models are trained on very downscaled versions of larger images. Even if high-frequency data was discarded in the original, that wouldn't necessarily mean that high-frequency was discarded in the downsampled version the network sees. It would depend on details of the data processing pipeline.)
To be clear, I'm not a neuroscientist and this is all just my understanding from the ML side!