There are several considerations that underpin my own thinking on social justice, but perhaps most relevant here is the concept of competing freedoms. It's easy to say that your right to swing your fist ends at my nose, but the debate over hate speech is much more controversial. In the case of personal threats or libel, we do have a history of recognizing that there are potentially competing freedoms at stake when it comes to certain kinds of speech. We can productively disagree on how to balance these freedoms - even at the extremes - but the caricature of wokeness pretends that I don't care about freedom of speech at all, while simultaneously dismissing that there may be competing freedoms worth considering. I personally believe that freedom of speech outweighs the freedom to live unencumbered by impersonal threats and libel - but I also acknowledge the damage that allows, and believe we should consider ways to minimize or compensate for it.
This applies to my example of sexual harassment, but in that case with what I will call "soft freedoms." That is to say, the hypothetical harasser does not have a right to work at that company, and the hypothetical victims do not have a right to not see a particular person. In a perfect world, a reformed harasser could return to work and get a second chance, and those they worked with could feel secure and safe. Given that we do not live in this perfect world, there is a choice in how to balance these soft freedoms. I think we might disagree on where to place this particular line, but probably not by very much. But the "anti-woke" would have you believe that I think that anyone who was accused of the slightest transgression should be fired.