Just recognize that (I'm going to go out on a limb here and assume) that, because you are a middle class white man, probably 90% of the reason you oppose "affirmative action" is because it negatively impacts your own demographic group.
Arguments for or against AA aside, this line of reasoning, where you invalidate somebody's argument based on what demographic group they belong to (white middle class males in this case), really seems like a simple ad hominem argument. You're no longer addressing what is said - only who is saying it.