"The more parents talked to their children, the faster the children's vocabularies were growing and the higher children's IQ test scores at age 3 and later. [...] [By age 4], an average child in a professional family would have the accumulated experience with almost 45 million words, an average child in a working-class family would have accumulated experience with 26 million words, and an average child in a welfare family would have accumulated experience with 13 million words. [...] Just to provide an average welfare child with an amount of weekly language experience equal to that of an average working-class child would require 41 hours per week of out-of-home experience as rich in words addressed to the child as that in an average professional home. [...] By the time children are 4 years old, intervention programs come too late and can provide too little experience to make up for the past."
The basic idea is that the best determiner of academic success is the number of words per day spoken to the child (listening to audio/tv doesn't work, and is generally thought to be harmful, especially before age 3.) Essentially by the time kids get to kindergarten, there is already a 2+ year achievement gap between wealthy kids and poor kids, mostly for this reason. In addition to being enormous on its own, this gap then compounds greatly because of tracking, which tends to be introduced in some form or another starting around age six:
"We found that first-grade ability-group placement can have persistent effects on children's achievement in school over a period of several years and may shape the expectations of children's performance held by significant others, such as parents and teachers. Whether these effects are instructional, social, or institutional, they are real, and they have implications for children's future schooling trajectories. [...] Instructional grouping may have the unintended effect of increasing inequalities in educational outcomes, largely by creating inequalities in educational resources and rewards." (I forget the author, but this is quoting in Riordan's book Equality and Achievement in the relevant section.)
Anyway I'd definitely recommend reading Meaningful Differences, in addition to being a great example of how science should be done in terms of robust methodology, it's also one of the most insightful and important scientific findings of the last hundred years.
The one thing I'd add though is that in addition to the total volume of language, which is the most important variable, there are a few other counterintuitive findings worth knowing about. E.g. reading aloud to your kids doesn't actually benefit their language skills, unless you purposely use certain linguistic constructs while reading to them. And there are other linguistic features you can use that are beneficial, that you wouldn't necessarily think to use without knowing the research, but I'll stop there though so as not to spoil it.