I think that's an unfair reading of things. As a counter-example, my kids are in the "honors" track at their public schools, have done accelerated math since middle school, and will end high school having done Calc BC as a senior (my son) and a junior (my daughter), as well as AP Stats. To me, that's "good enough" for kids who aren't particularly interested in higher maths. That said, the kids are intelligent and if we pushed they'd surely move faster.
But instead we throw the majority of our discretionary time at sports. It's entirely possible both kids will have opportunities to play in college, likely in D1 programs (track & soccer, respectively). This isn't because either probably have the raw genetics of [pick a superstar], but because they have been practicing each one 8-10hr/wk since they got to about 6th grade on top regional teams & clubs. If we throw the amount of time, money and effort we put toward sports (on average, probably $6000/yr for soccer and about $3000/yr for track/XC) toward academics -- or any other extracurricular interest -- surely we'd see above average performance in those things, too.
Fwiw, I'm a big proponent of public schools for many reasons, but one of them is that the lower academic stress on top students (at all but the most elite schools) gives them more time to explore well-roundedness through hobbies, sports, jobs and other activities.