The problem isn't physically abusive partners alone. There's more to it than that. If dating a jerk or a crazy person or even a physically abusive person made someone "permanently damaged" there'd be almost no one left.
The main issue that is gendered is that women have more opportunities, when they're young, to involve themselves with really terrible people and for things to go seriously wrong. We're all bad judges of character when we're really young. The typical 15-year-old boy isn't any morally better than a 15-year-old girl, and would probably make choices that would destroy him if given the chance... but typical 15-year-old boys are considered unattractive (even to girls of the same age) and don't get those opportunities. I was a fucking idiot at that age, but no one wanted to be a fucking idiot with me. For women... it's a different story. The opportunities to make really bad choices and ruin yourself, if you're a young woman, are quite available.
I don't think it's even sex itself that does the damage. I used to think that, but I think casual social contact is what erodes the person's values. Being around bad people makes a person bad. What makes college casual sex evil isn't the sex itself (the sex is vanilla) but the acquisitive, manipulative, and dishonest culture around it. I have no problem with swingers or people in open relationships or above-board promiscuous adults, because there's nothing gross or wrong about sex itself. But the culture around college casual sex (the alcohol abuse, the focus on man as conquerer and woman as defeated, the "gray rape" that isn't "gray" at all, the fights, the "pickup artist" nonsense, and the reward of bad behavior in both sexes) is pretty disgusting.