Consider the opposing argument - groups have no responsibility for anything.
Does that seem like a reasonable position?
Consider an expansion of the argument - are there still situations in which one group consistently discriminates against and abuses another?
People are already in castes, defined by their lineage and physical (and economic) characteristics.
You don't break down the walls between castes by pretending they're not real.