Even sexists.
I think the more prevalent problem is the underlying sexism, the institutional bias, the reluctance to speak up, the fear of being judged "the token" -- the fact that a man can think it's appropriate to put his hand on a woman's knee unsolicited is just a blatant, ugly manifestation of the underlying sexism.
I know this post had a little more nuance, but it's hard to come away from it without the impression of: "Women aren't on speaker panels because they're afraid of getting molested". Maybe that's the case (and if so, we as a society have serious issues)...but there's at least an equal problem of women just not being considered or being searched for. And as that problem persists, then women continue to become such a minority at these conferences that the aforementioned creep feels perfectly comfortable with invading a woman's personal space.