That is probably mostly because the population argument is usually used by people in rich countries in order not to have to change anything, despite the fact that on average, a 10-person family in rural Africa emits less CO2 than a single individual in the US.

I agree there is probably no good reason for anyone to have more than 4 kids (at most, probably just 2 to go for a manageable population decline) but lifestyle is simply a more important aspect of the equation.