A more historically recent but analogous scenario is the social and racial structure of Latin America, wherein those with more white European ancestry tend to have higher social status compared to those with more indigenous ancestry. Most people have some of both, but in the logic of such systems, proportions of ancestry matter, and phenotypical visual appearance is used as a proxy to gauge this, whether in South Asia or Latin America.
Also, the connection between South Asia and Latin America's rationalization of their social systems is not coincidental. The term "caste" itself is derived from the Portuguese word "casta" used by traders in South Asia to describe the prevalent social structure they encountered. They (and the Spanish) quite deliberately imported this model to their colonies in the Americas.
They are among people who want to discriminate on the basis of race.