Fairness (in the sense of trying to create a level playing field and make sure nobody has unfair advantages) is important to the extent that something is a competition.
This means people who see education as competition will care about fairness - either they will want things to be truly fair or they will want any unfairness to be in their favor.
So from this point of view, special classes for kids who are already doing better can be seen as an unfair advantage (especially because there's plenty of real bias involved in determining which kids actually end up on those classes).
I think we need to fix the biased selection process and make it more possible for more kids to benefit from advanced learning opportunities, especially kids whose parents don't have the resources to take them out of public school to do something more individualized.