Observing a bias, quantifying it as best as possible, and deploying countermeasures in corresponding degree to reduce its impact, is not racism.
But if you believe anything which assigns admission/hiring/recruiting scores on any basis other than objectively-measured individual merit is "racist", you cannot plausibly be in favor of current hiring, recruiting or admissions practices, and have far larger fish to fry than the relatively small number of people whose admission/hiring is solely due to affirmative action.
In all seriousness: look up the ratio of "legacy" admits to, say, black students admitted at top colleges, and then remind yourself that "black" is not synonymous with "affirmative action admit", and then ponder which of "legacy" and "affirmative action" would cause the greatest overall change toward fairness if eliminated. Then focus your efforts accordingly.