But people who believe in "races" don't think that it's just a matter of skin color. They think that skin color (or whatever other feature) is a marker of some deeper difference. That's where the mistake lies. Otherwise racial differences would be of no more significance than, say, differences in hair or eye color.
Racial categories are nowhere near as objective as you suggest. For example, Italians used not to be considered "white" in the US. Many Americans who self-identify as "black" would not be considered black in many parts of Africa.