You can certainly ostracize people who don’t deserve to be ostracized, but it doesn’t follow from that that all social ostracization is inherently bad. Freedom of association and freedom of speech can be used for bad ends as well as good, like most freedoms.