I think your hypothesis is mostly correct, except for the idea that the men who are having more sex are “tremendous specimens”.
I don’t think they need to be especially attractive or powerful. It could be cultural or psychological differences that make some men put more energy into their pursuits. For example, compare Jimmy Carter to Bill Clinton.