It makes for a lot of miscommunication and drama.
It very much sounded like they were both comfortable with their relationship. Granted, this was his writing many years after the fact.
I don't think the claims that he was a womanizer are relevant or fair. People say that kind of shit about MLK to discredit him. Similar claims against Feynman don't discount his love for his wife.