The argument based on collective guilt assumes that there are no differences between whatever is labeled race. Racists may not agree.
Moderates stepping into arguments about racism made by extremists (on both sides) might not realize how bigoted the idea of "racism" itself is. Those on the left argue that the "oppressing" race should feel guilty, not just that no one should feel pride in their race.