The way I see it, one group is arguing from a place of historical hardship, and the other is arguing from a place of privilege. I don't mean someone growing up poor in Appalachia as being privileged simply because s/he is white; to me, that historical background definitely counts towards raising diversity. I'm just trying to put myself in other people's shoes. It's not necessarily about race or ethnicity or recentness of oppression, it's the compounded interest of those oppressions and inability to rise out of that enforced debt. Our institutions have to factor that historical indebtedness.
Unless you really put yourself in another person's shoes, I honestly don't think this discussion will go anywhere. I mean, imagine growing up poor in a single household family, going to lowly ranked high schools, being disproportionately more likely to be stopped over and imprisoned for minor offenses, and getting questioned about academic abilities constantly.