That's your subjective opinion though. There are studies that report very high rates of false accusations. You may decide they aren't good studies or you don't believe them or whatever, but if other men don't write these things off immediately, it's not hysteria.
Many of these recent allegations would certainly not have been taken seriously in earlier times: the bar for a report to be considered false has been raised to stratospheric levels. Once, if a woman claimed she'd been sexually assaulted by a man, the fact that she didn't bother to report it, voluntarily chose to work with that man for years afterwards, was routinely photographed in public looking happy with that man, tweeted about what a great time she had that night, wrote him thankyou notes etc would all have been considered fatal to her case. Nowadays many people seem to believe that it's impossible for women to lie or exaggerate such claims and thus ignore any evidence to the contrary.
In such an environment, it does actually make a lot of sense for men to just avoid working with women. The norms of justice can't be relied on, so total exclusion is all that's left. Not that I'd do it. But I understand those who do.