But yellow isn't necessarily just a spike at 569.356... There are plenty of other combinations of frequencies that together will stimulate the green- and red-cones enough to create a perception of (nearly?) the exact same yellow.

If your ear had only 3 types of detectors which only detected 3 specific frequency distributions within about half an octave but could locate stimuli within your field of hearing with pinpoint accuracy, after a lifetime of using that equipment you would probably be able to make relatively fine distinctions in pitch in that very limited range.

Instead, the human cochlea contains thousands of little pitch detectors spread over 10 octaves, and the perceptual architecture and typical training built around it is designed to detect relative pitches (e.g. noticing the difference between two different people’s voices more strongly than the absolute frequency of the fundamental pitch of either voice).

Eyes and ears just have fundamentally different physical mechanisms and we make sense of visual and auditory stimuli in fundamentally different ways. They are not really directly comparable.

In both cases, however, our perception is strongly context-relative.