I generally agree with you, and I like that it doesn't harp on this aspect when talking about his life, but doesn't pretend it's not there either.

I am not familiar enough with Feynman's life to hold a particularly nuanced opinion, but it seems like just that one line intentionally taken out of context seems to be the issue. A one-line summary of the same chapter in the book could show that he was a womanizer without taking a particularly inflammatory quote out of context.

In an unlikely coincidence, that particular chapter of Surely You're Joking, Mr. Feynman! was one of the few that I had read.