I do not know the literature well enough to cite anthropological studies. But as far as I understand it, this is a relatively uncontroversial claim within the field. Jared Diamond's article is a good introduction:
http://www.ditext.com/diamond/mistake.html
Edit: As others have mentioned below, the main evidence is a decline in the height, bone density and tooth health of skeletons of pre and post neolitic humans.
Average lifespan also declined, from 33 --> 20 according to Wikipedia. (I don't know if the studies cited are good sources)
Keep in mind that these lifespan figures include infant mortality. This explains why they remain so low up until very modern times.
http://en.wikipedia.org/wiki/Life_expectancy#Life_expectancy...