To examine this "unfairness", let's imagine it at its most extreme: a society in which divorcees are so stigmatized that it's practically impossible to ever re-marry; in which children of divorcees are likewise stigmatized; in which victims of child-abuse are further victimized by a society that considers them potential "snakes-in-the-grass". Do we really want to live in such a society?
Obviously I don't think you were advocating this. But it's a thought experiment which demonstrates a classic class of problem: what's good for the individual isn't always good for society, especially taken to extremes.