Here's where your thinking goes askew, you can't simply draw a boundary around a subset of people and declare that an agentic thing. Groups of people don't have guilt or automatic responsibility, only individuals do.
Thinking of very diverse groups of people as single entities is how you get sentiments like "Muslims did 9/11 and they must pay" without considering that the tendency-towards-9/11-ness might not carry over to the entire set of "Muslims". Less than half of Americans were even alive in 1971 and no one is alive from the days of US slavery.
Thinking "those who have inherited benefits due to negative treatment of African Americans should transfer wealth to the descendents of those African Americans" is a separate idea to race based affirmative action. Race based AA would see the children of a pair of Ukrainian immigrants put below the children of a pair of Ethiopian immigrants, even though neither group has anything to do with slavery.