Everyone has plenty to fear. Any sexual harassment/rape accusations (no matter how untrue) are incredibly damaging, it's very biased towards the accuser.

The people who actually do this shit are scumbags, but there's plenty who will take advantage of this when things don't go their way, and reputation damage is worse than physical for a lot of people.