If there is any way for genetics to influence family size, it'll quickly spread. Cultural factors and resource constraints dominated for millennia but we'd now expect things to go the other way.
Religion isn’t a veritable trait, but the correlation is strong.
Partly heritable
The part where you inherit big enough house and other assets to make it feasible
If humanity survives then this trait will emerge and become dominant.
Humans are part of a lineage that's survived for hundreds of millions or billions of years. We've already had that time. The pressure is simply more complicated than "have big families whatever the cost" and humans have agency.