Color is overrated in a world where most are colorblind. Red means danger or stop because you have seen a lot of stop signs. In a world of colorblind people, stop sign colors are chosen by the colorblind, based on something like "is this a cheap metal" rather than design language (which no longer exists for color). Instead we would use reflectivity or something to indicate danger. As to whether or not the varying-colored stop signs leads to a statistically significant number of color-sighted people blowing through them it is hard to say, but does seem a legitimate safety concern.

And this is the situation in music; keys are chosen based on the technical details of the musician rather than a design language. Meanwhile, we do have a design language based on relative pitch (e.g. a 5/7 chord "leads" to the tonic). Of course you can see that the same way you can see a purple stop sign, but you may lack some of the pattern-matching intuition of your peers.

As a musician with relative pitch and the child of a musician with perfect pitch, I have watched them get puzzled a lot navigating a design language that is very obvious to me. Perhaps you are a better musician than they are, or perhaps you avoid areas where the design language dominates. Either way, I would not trade places with you, unless the rest of society did also.