Which is similar to the reason why blacks haven't bounced back from slavery that "ended in 1865". The started from less than zero. Not only poor (immigrants also arrived and started poor), but also seen as dangerous/inferior etc and having to face Jim Crow laws etc, segregation (up until the 70s), redlining, poorly funded school districts, etc.
That held them down far longer than most other races, e.g. asians, jews, irish etc had to face (with the posisble exception of latinos), and never allowed them to build enough "familial/class privilege" to sustain mass movement to middle class/upper middle class.
What makes me a bit sad here is that there is probably much more scientific talent out there that we're not tapping into by not providing good study conditions for everyone, and this is something that we must change.
It is true that being in a good lab helps a career, but these are nepotistic organizations. Join one as an undergrad to get experience then apply to one as a grad student or a paid research specialist. Ive seen that 100 more times than family members of colleagues.
The small number of families seems more likely attributable to the small size of the enterprise and the difficulty of the subject matter. It's very hard to have a good idea and equally hard to publish it. I don't know of anyone in pure math who decides about the value of a piece of work based upon pedigree.