Possibly, but to fair I don't think there is a reason. The shire is similarly inexplicitly underpopulated when hobbits have been peacefully having large families there for 1400 years!
After all, magic requires power, sometimes quite large amounts, and no power source is to be seen. This means it's either beamed, or otherwise dispersed through the environment. This could have negative implications for health and fertility.