What mattered is the parents of the poor school worked 2 jobs and had no time for the kids. The parents of the rich kids had stay at home moms and tutors.
Depending on your belief system there would also be very different genetics between the two groups.