You're... not right.
More like out of 5 men, on average only one would have any descendent whose genetic line survived until today. But he would have a lot.
Now, some of the other 4 might have had children, but they might have dies young, or in turn not have had any descendants whose genetic line is still around.
Of course, the same would have happened to women, so maybe out of 15 men and 15 women, 5 women and 1 man have genes that are still around.
Also note that, unless I misunderstand, this only applies to single gendered lineage. That is there must be an unbroken chain of male ancestors for the male genes, and female ancestors for the female genes. If one of the men without surviving male genes had 3 daughters who went on to have lots of kids, his genes wouldn't be seen in this data, even though his other genes are with us.
Having gone through that line of thought, I wonder if we're looking at some kind of "compound interest" effect here. I mean, if the male-only gene has 10% less chance to survive a generation, does it add up to much bigger and bigger numbers after hundreds of generations?
I'm not sure if that clarified much...