I think there are too many extreme exceptions to this correlation for it to be anything but a correlation. For just one among many, look to any of the countless religious groups within developed countries. These range from Muslims who continue have children at a healthy rate, to the Haredi in Israel who are popping out 6 kids per woman on average. Even Africa's blushing there.

The obvious and logical explanation, which fits seemingly all data, is simply culture. An obvious one is extreme consumerism. Extreme consumerism is going to drive big economies which makes healthcare widely available, driving down infant mortality. At the same time it's also going to drive people to spend their lives trying to earn money instead of raising families, driving down fertility.

Without culture playing a largely dominant role it would also have been much harder for countries to artificially manipulate their fertility rates, as Iran and South Korea both did - and later came to severely regret. I'm leaving China out of the examples as they took more of a legal than cultural approach to try to change their fertility rates.