1. The player/whore dynamic. There are male whores and female players, but usually the woman is going to be presumed to be the conquered, the pwned, the slut. This double standard is not going to go away soon.
Empirically, the vast majority of people who participate in casual sex have low self-esteem. In men, this type of activity is (perhaps wrongly) viewed as caddish but a not-entirely-repulsive way of maintaining one's confidence between relationships. Society is less forgiving of women who do it (again, perhaps wrongly).
2. Casual sex makes it difficult to form intimate relationships later in life. Future partners will view it negatively, and the psychological consequences of casual sex are severe and generally undesirable. The "wires" for emotional intimacy and sexuality are closely intertwined for most people, but once separated, they don't go back together.
3. It turns physical intimacy into a commodity, traded on a highly efficient* market where the media of exchange are social status, physical attractiveness, "game", and "for the story". (* However, "inefficiencies" such as love and devotion turn out to have enormous subjective benefits, for people of both genders.) So, I'd phrase the question as thus: why is prostitution considered anti-feminist? The only difference between casual sex and prostitution is that, in the former, the medium of exchange is social status, and in the latter, it's money.
4. Casual sex represents (admittedly contained) social regression. In primitive, pre-monogamous societies, a few men (alpha males) have many wives, a larger set (beta) have one or two wives, and most (gamma) have none. Surplus loser males must be sent off to war to acquire more resources (and therefore, more women) or they will stage a revolution against the alphas. (In pre-monogamous societies, 1 in 3 men will die at the hands of another male, and murder often raises a man's social status.)
What does monogamy do? For one thing, it reduces the reproductive stakes involved in social status, making it simply not worth it to kill another man in order to improve one's position. Second, social status must be redefined; instead of seeking many wives, successful men are encouraged to find one very desirable partner. This creates an incentive for him to treat her well, because a wife in this arrangement has a lot more power over her husband than if she were one of 8 wives.
The casual sex market is a pre-monogamous subculture, and although it hasn't led to full-on social regression, I would not be surprised to learn that it's a major contributor to violence and rape. There's a reason fights break out at nightclubs... and alcohol ++ is merely a catalyst, not an agent.
A few fringe feminists support casual sex out of a reaction against the 1950s "traditional family", which they view as an abomination. The traditional family wasn't the best regime for a lot of people, but there are practically as many family structures as there are actual families, and most modern families are much closer to equal partnerships than "traditional" families. In any case, replacing the family with no-holds-barred commoditization of activities that were once called "physical intimacy", and calling this a "feminist" advance, is just balls-o'er-the-bridge lunacy.
For the record, I'm utterly liberal. I'm appalled that people in this country want to ban same-sex couples from getting married, and I'm generally a social libertarian. However, I'm also a pragmatist, and the fruits borne of casual sex are detrimental to society. I would never support a law banning casual sex, as it would be impossible to define, but I'd prefer that this class of activities fade into the background as behaviors to be avoided, rather than being glamorized by shows like Sex and the City.
( ++ Later edit, 3:32 pm. I would actually classify the alcohol abuse observed in the bar/club scene as case-in-point of sexually driven violence. Excessive drug use is, arguably, violence against oneself, and the case could be made that "social" alcohol abuse is often borne out of sexual frustration.)