Life expectancy is better today, but life expectancy as a hunter-gatherer was probably better than you think. [0] suggests a modal adult age of 68-78 years for hunter-gatherers. Infant mortality is indeed way down, and that's a big point in our favor (and the biggest driver of the life expectancy difference).

Infectious diseases, though, are very much a product of agricultural society, as is starvation/famine. We might be living through the century (or possibly, have just lived through the century) where our technology finally enables us to fix these two problems caused by our ancestors' conversion to agriculture.

[0] http://www.anth.ucsb.edu/faculty/gurven/papers/GurvenKaplan2...