Specifically, have schools tried to partition students into courses based on achievement level, regardless of age? There would be the Math 1 course that might have 1st, 2nd, and 3rd graders in it, and Math 2 might have brighter 1st graders, etc.
The problem, I think, is not that kids can't learn (in fact the amount neuroplasticity in young children is simply off the charts, to the point where 6-7 year olds who can play really good violin are not something special), but is that the educational system simply does want to go there, to increase the levels of math taught in schools.
(Born, raised, and educated in California)
However, one reason it might be avoided is due to the potential negative effects tracking [1] strategies may have on some students. In my public school district, any form of tracking at the elementary level was difficult to implement due to pushback from parents, even in situations where the educators felt it was beneficial.
A decent fraction of the kids learned math at home, making it hard to assess how well the school was actually doing.