I don't see any evidence that the heritability of desire to have large families (or the heritability of factors linked to not accidentally having large families) is anywhere near as strong as the effect of socioeconomic trends which have decimated the cultural and rational reasons for having large families in developed countries. Looking at a different aspect of heritability, modernity did far more damage to the intake of celibate religious orders and celebration of virginity than aeons of selective removal from the gene pool of those especially vulnerable to the idea that chastity was a virtue.

Populations of people that think birth control is a good idea and are quite happy to defer their children until later in life still reproduce in sufficient numbers to remain the majority in developed countries, especially assuming that many offspring of reproductive fanatics will end up "defecting" and behaving according to the prevailing social norm rather than their parent's preferences or inherited instincts.

Given the continued reproduction of people who think birth control is a sensible idea, women should have careers and more than three kids is a headache, I think heritability of desire to have large families would have to be almost deterministic for us to start to worry about hyper-fertile people taking over.