If a woman bears many children throughout her life, you can still have a situation where the first pregnancy occurs much earlier than today's average yet the overall conception average is roughly same as nowadays.
I bet 100k years ago, some women were pregnant yearly, from teenage to menopause.
If men were begetting from a young age 100k years ago, and most of their children died until they were older and more experienced, those early children are lost from the DNA record.