Ability based learning within age groups makes perfect sense, but assumes a sample size large enough to make meaningful groupings at the high and low ends.
You might submit conferences as a counterexample, but I would submit them in favor of my argument.
A preferable alternative in my mind would be a classroom where ALL work is done independently. The teacher then calls each student to their desk individually to see what they have learned and what they need help with. They can then spend 10-15 minutes individually with each student per day. Fast students can get pushed ahead and slow students can get the help they need instead of being forced along with the rest of the class.
So for one it takes 6 months to learn something, for another - 3 years. And in such systems it's fine - you got promoted because you have learnt something, not just because you got one year older (and haven't failed utterly).
And again, for the same "level" of subject there can be two speeds - so no risk of smart 8 yo kids in the same class with slow 11 yo.
This system works out very well for bright, academically inclined students, sure. But it sets a structure of expectations that don't motivate what could possibly be great talent, but we wouldn't know because they didn't bubble in the right answer.
The fascist premise of "let's separate the most able from the riff-raff".
As if society should be a hierarchical pyramid of top dogs and underdogs.
This doesn't mean that ability as a differentiator is bad--I think that it's a great idea. What it does mean is that consistency is absolutely crucial when such an idea is implemented; I was in an environment where excellence was encouraged, and that led to a lot of self-motivation. Not all children are going to be in the same situation, and that could be disasterous.
"The Art of Learning" by Josh Waitzkin tackles the concept of the environment which best facilitates learning and personal growth, and touches on the idea of ability-based classes as well. It's a great read for anyone interested in such.
I doubt anyone could argue against things like AP classes, etc. in high school, but 5-year-olds?
Which is why so many "proteges" tend to burn out, I think, or have emotional problems as they age - simply for lacking that time to develop with their peers.
Don't just think about slightly tweaking the system we have today. Think about the system after it finally embraces computers as a transformative force, instead of a very silly expensive way to issue multiple choice tests. We're going to free up teachers to spend their time on the stuff that actually requires a human, instead of bogging them down with so much automateable work that they end up hardly any more effective than a machine, what a surprise.