This paper is under strong discussion, but it argues that humanity had a period (between around 930,000 and 813,000 years ago) where we had a severe population bottleneck with about 1280 breeding individuals.
No wonder humans seem to do their best when things are the worst