Infant mortality has everything to do with medical care and very little to do with the diet. That's why infant deaths should be thrown out when talking about the diet.
If I have 3 people, one died at age 0, one died at age 50 and one died at age 60, the average lifespan is about 36.5. Clearly misleading. I don't think people realize just how high infant mortality was prior to the advent of modern medicine.
If you have 5 people who are born on the same day, 4 of which live till 80, 1 of which dies shortly after birth, the average lifespan will be 64 even though 80% of the people lived until 80.
His point was that most of our increase in average lifespan can be attributed to the reduction in infant mortality rather than people living to higher ages.