Desire to have children has been under selection only since the demographic transition. Most of the world has less than two generations of selection, most one, some none. As such we would expect to see a lot of variation in desire to have children. The only group I’d be willing to bet money have notably higher desire to have children than other human populations are the French because that’s where the demographic transition started. For everyone else on average we should expect more or less the same mean and variance.
Then the environment changes in one, or in two ways. People move to another country, or not, and their culture gets exposed to modernity/the demographic transition. If they move to another country the culture they’re surrounded by will change, which will change when their children think it’s appropriate to marry, how many children they want, etc. It will do nothing to their genes, including their desire to have children.
But all over the world the selection is ongoing. As time goes on the proportion of the population that wants lots of children increases and so do the number of large families.
Genetic and sociological explanations can both be true at the same time. Mormons have more children than average for sociological reasons. They value and honour child rearing and family life far, far more than normal Americans. People whose parents had 6+ children also have more children than average, possibly partly due to role model effects, but mostly because they like children. There’s nothing incompatible about these explanations.