I agree, but I think you too underestimate how random it is. The impact of being "elite" is probably tiny compared to the impact from chance, and at least historically, those in the "elite" genetically speaking weren't the ones at the very top. Kings live risky lives, as do sons of kings. A successful merchant able to provide a trades education to each of his 8 sons, would probably have far more Y-line descendants than a contemporary king.

And in the last couple of hundred years, even that "elite" (middle class) had a huge "disadvantage" in getting Y line descendants: they were early into the demographic transition, when everyone started having far fewer kids. Going from 11 kids to 3 kids 2-3 generations earlier meant that a lot of our elites (in the traditional sense) basically missed the population explosion in the last 150 years!

It's much smarter to ask, "where were they when the population explosion happened" than to prematurely imagine harem slaves.