The neolithic revolution also caused a bottleneck in the diversity of the Y chromosomes.
9 out of ten men at the time didn't have the opportunity to reproduce, while the happy fews were highly polygynous.
This can't have happened peacefully.
When population growth is effectively 0, the size of the male lineage of the average man is approximately 1 in each generation. But random variation means that the male lineages of most men don't survive that many generations.
Now imagine a stratified society, where elite men have more children than commoners and social status is largely hereditary from father to son. The average elite man will have >1 sons surviving into adulthood, while the average commoner will have fewer than one adult son. Repeat that over many generations, and lineages that started from an elite man will likely spread, while lineages starting from a commoner will likely go extinct.
And in the last couple of hundred years, even that "elite" (middle class) had a huge "disadvantage" in getting Y line descendants: they were early into the demographic transition, when everyone started having far fewer kids. Going from 11 kids to 3 kids 2-3 generations earlier meant that a lot of our elites (in the traditional sense) basically missed the population explosion in the last 150 years!
It's much smarter to ask, "where were they when the population explosion happened" than to prematurely imagine harem slaves.