If the colors were the same thing as the electromagnetic frequencies, then the same electromagnetic frequencies would be the same colors, by definition.

They aren't. For example, put a card of color A in front of a background of color B; now move it in front of a different background of color C. You will experience color A as being a different color (especially if colors A, B, and C are chosen to maximze the effect).

The electromagnetic spectrum returned by card A isn't different, but the color perceived is. Thus, electromagnetic spectrum is "out there", but color is "in here".