1. Breaks down especially with the existence of mixed race people. Did you know that there was a term for Americans born with 7 European great grandparents and 1 black one? Many of those were born into slavery and treated horribly for being black.
Which touches on 2... But also, the Irish were not considered white. I've seen horribly racist British illustrations of the 19th century comparing Irish people to Africans. In that century in the United States, Italian and eastern European immigrants were also not considered white.
When you consider these ridiculous corner cases, and how their perceptions have changed over time, it makes you aware that the categorization itself isn't really based on logic and reason or facts, even when generalizations hold up "better". Yes, for the most stereotypically appearing people, we'll probably correctly guess their ancestry a lot of the time based on things like appearances. But it falls apart pretty quickly. You don't need to get into extreme outlier cases as above for you to miss something when you treat racial categories as unambiguous facts.