Perhaps all that magic is reducing fertility rates?

After all, magic requires power, sometimes quite large amounts, and no power source is to be seen. This means it's either beamed, or otherwise dispersed through the environment. This could have negative implications for health and fertility.