People didn’t live long enough to have cancer back then.
If a quarter or half of your population is dying at age 0, then another quarter or half must likely be living into their 70s for the mean age to end up 35.
Well not really age 0, rather between 0 and 5.
The rest of your point stands.
Life expectancy of children and young adults has increased significantly in addition to the reduction in infant mortality.