And what does this have to do with colors being spectral? If you look at the cie chromaticity diagram you can see that yellow and cyan have wavelength corresponding to them while purple does not.
This is important because privileging magenta does not generalize to higher-dimensional color-spaces, and therefore impedes to our understanding of phenomenology.
E.g. a "yellow" wavelength is only as bright as its decomposition into red and green. The fact that red & blue aren't adjacent is a red herring.