450 THz electromagnetic waves are unambiguously red light, you can nitpick about the shade of red, how people perceive it, etc... but it is red. It may look differently to colorblind people, or dogs, but it is red. The electromagnetic spectrum of frequencies just lower than red is called infra-red, don't tell me that physicists don't talk about infrared.
If you look up some colors in Wolfram-Alpha you will have precise ranges of frequencies, I am not sure they are officially defined ranges, but colors, at least pure colors correspond to physical quantities.
There is some overlap with biology, and there are some colors are special (brown, purple, ...) and can't be defined by a single frequency. Then you can introduce color perception, and how very real and physical red photons are perceived as red, why a combination of red and green photons are perceived the same way as yellow photons and how spectrometers reveal the difference, and how there are no purple photons.
Composite colors are a bit more complicated, you need to add a bit of math (namely, a color space) in order to formally define them from physical quantities, but saying that it is not a physical concept is going a bit far.
No need to make physics more abstract than it is. Because what's next? Temperature is not a physical concept because we don't all agree on how hot something is? Like color, temperature is a well defined physical concept that is based on our perception.