It's kind of hard to figure out what to do. Harassment is obviously something I would speak out against. But when we define "harassment" to be whatever a minority member in earshot finds offensive things become murky. Perhaps I should constantly poll nearby women at a conference - "ma'am, has anybody said anything recently which you find offensive?"

There is such a thing as unacceptable behavior. But the measure of that is not whether some member of a protected group was offended. Nobody has the right to never be offended in a public place.