Could be that there is a kind of developmental switch. If life is tough as a child, it makes sense to speed up development to improve odds of survival. If you find yourself in a safe, loving and caring place, then you have the option of spending more time and energy developing the brain.

> life was hardship and adversity

Not necessarily. Many argue that hunters and gatherers lived a much better life than during most of recorded history. There is plenty of evidence for this. It's entirely possible that a child would find itself in a small, well functioning group of people, without any major adversity during childhood.