I know he was somewhat rakish before he married his third wife, but in what manner was he terrible to women? It takes two to tango, so his rakishness doesn't seem to imply sexism.
And partly it really does feel like Feynman literally thought less of most of the women in his life. (On some level, that's par for the course: Feynman was an asshole to just about everyone.) Just for instance, the conclusion of that "You just ask them?" story (from memory, again) includes the observation that female college students are "no better than" women who hang around in bars, whom he had already called "whores" (and seemed to mean it). See also [0], again.
Some of this is a reflection of the era when he lived. But it's a particularly vivid reflection, and it was a pretty thoroughly sexist era. When Feynman is held up as an ideal without keeping that context in mind, it can tend to propagate the shortcomings of that era into the present.
It does indeed take two to tango. But nothing about that guarantees that both dancers are equals. When I tango, just for instance, tradition insists that as a man, I get to lead, and my partner's job is to do what I tell her to as gracefully as possible. It's stretching the metaphor to apply that directly here, I know (though it is unquestionably a manifestation of our society's invisible background sexism). But I suspect that most of Feynman's many lovers did not think less of him in the ways that he very explicitly learned and practiced thinking less of them.
[0] Just for instance, in one of his stories, Feynman was clearly delighted by the description of a local woman in Brazil as "a sleeping dictionary", since he didn't speak Portuguese well.