Malnutrition and serious physical abuse aren't the only environmental factors with a known significant impact on intelligence, and nutrition (particularly childhood nutrition) patterns with a serious adverse effect on IQ have been identified in research focussed on first world areas of study; they certainly aren't rate enough to be insignificant sources of variation in the first world.