Like okay, you get X points if you're this race, Y points for that. Z points if your parents were poor. Or if you grew up in these bad zip codes. Or if your dad was gone and mom was an addict. Whatever.
As an Asian American of relatively privileged (middle class) upbringing who went to a state college, I often found it unfair that many of my desperately poor white peers worked their asses off their whole lives, despite minimal support from their parents, to get into college on merit alone. Meanwhile for me, my admissions counselor handwaved away all the entrance requirements (my GPA was low, I didn't have the pre reqs done, etc.) and admitted me on the spot. Years later I'd find out that I was part of their zip code based recruiting program designed to get non whites into the school for the benefit of their diversity quotas. In California they already weren't allowed to use affirmative action due to Prop 209, so they just used a geographical proxy for race (finding zip codes with high non white ratios to recruit from).
I didn't deserve that spot at all. I never worked for it, I never suffered for it, my parents didn't much either. I just happened to benefit from policies meant to protect Black and Hispanic people, at their expense, while simultaneously throwing white people under the bus. It was pretty unfair all around.
I get that as a society we want to help give people a chance to escape the circumstances of their birth. But skin color alone is an awfully broad brush that paints only a vague picture of who that person is and what kind of adversity they may or may not have overcome. I wish we looked at it with more nuance is all I'm saying.