If we are thinking in terms of democratic societies here, then certainly the (utilitarian) point of education should be to raise the bar especially for the "average kids," shouldn't it? Don't we want a better average in a society that we all operate together? That's to say nothing of the richness in life that education can provide...
The utilitarian metric could also show that the average moves the most, and society benefits most, when those with the most potential can realize it. It's an interesting question. Either way though, I think it should be a basic constraint for an education system. Basically it's the same as saying classes should never be too slow for their students.
Not clear how the average is going to be "better" in any way by forcing them to.
> the richness in life that education can provide
The danger to deny the richness to those who can appreciate it.
Not in any society on Earth yet.