You know very well that statistically speaking, the argument for affirmative action is in one direction, your own personal acquaintances notwithstanding.
On the one hand, the average black person in the US is more disadvantaged than the average white person. That much is true. If you meet a random person, and they are black, statistically they are more disadvantaged than if they were white.
On the other hand, I'm fairly sure it's the case that for almost any given threshold of disadvantage, there are numerically more whites below it than blacks, simply because there are more white people in the US than black people. So if you are a university with hypothetical applicants drawn randomly from the population, the majority of the disadvantaged applicants in the pool are white, for most thresholds of disadvantage.
A system that treats disadvantaged whites, who are quite numerous, as if they are advantaged, based in the colour of their skin, is incredibly unjust, and creates deserved resentment. Frankly, even if disadvantaged whites were less numerous it is still incredibly unfair.
AA based on poverty and socio-economics directly, rather than the colour of one's skin, would be more just (and is more how things are done where I'm from).
Person 1 says, don't generalize ideas of privilege since it isn't fair to certain individuals that stand out as edge cases.
Person 2 says, the statistics say the majority of the cases the generalizations are consistent.
This conversation is a category error.
A just and fair society is like a fractal - it can't be fair "overall", it has to be fair at every level as you zoom in. There are no shortcuts here; with them, at best, you can create a society that pretends to be fair.