Not that long ago, when my parents were children, there was no such issue denying them the ability to use certain bathrooms or drinking fountains on account of their race, social construct or not, let alone attending the best schools. Thankfully, affirmative action was created by the time they went to college.
Your point is not a bad one. Race is not rigorously defined. And yet, there are large groups of people still suffering significant disadvantage along racial lines, for historic and ongoing reasons. It's not possible to address this in a race-blind way.
In practice, affirmative action policies work essentially on the honor system.