Life expectancy is a weird one, not going to lie.
All the numbers are wrong from some perspective.
You mention a good one. The fact that life expectancy was so low in the past was due to infant mortality. If you lived to like 15 (IIRC), your life expectancy got much longer. Like 60s to 70s (once again, IIRC).
And that even holds today. The longer you live, the longer you can expect to live. If you're 70, you have a life expectancy of 14 (male) to 17 (female) years. If you're 80, you have a life expectancy of 7 (m) to 9 (f) years. If you're 90, you have a life expectancy of about 4 years.
And all of that seems kind of weird on the face. If I'm 70, I can expect to live 14 years on average. Which takes me to 84, which is greater than 80. But if I'm 80, I can expect to live to 87, which is greater than 84.
But it's the nature of averages. When I'm 70, I'm being average with other 70 year olds. Some of them die at 71, some at 93. Every year, more and more people drop off. If you make it to 80, all those who didn't make are no longer relevant.