Isn't your reasoning backwards?

We know that improvements in living conditions lead to reduced fertility. And climate change will worsen living conditions, not improve them. So if areas with poor living conditions have high fertility now, why would even worse conditions (due to climate change) reduce their fertility? Sure, the outcomes and quality of life of the children will be even worse than it is but I can't see how that will impact rates of reproduction.