The sex drive is the most self-important part of the mind. It's the only part proclaiming itself as "indispensable". The feeling in our pants is definitely signalling to us "if I wasn't here feeling these urges our species would be dead by now."
There's still generic variation to the point where for some people this isn't a drive. Sure, it's a trait which self-selects itself for removal, but it's likely polygenic, which means it is much more likely that it can persist across generations despite being selected against. A lot of these things are non-intuitive, like how altruism can persist across generations despite being similar.
This is why high IQ is evolutionarily disadvantageous after a certain point. Probability of mating declines with increasing IQ.