This assumes that advanced courses harm students not enrolled in them. I don't see why that should be the case, and would like to see some evidence for this dichotomy you presented.
If you have 3 teachers, and they A) each teach classes that are composed of 20 regular and 10 gifted students, or B) 2 of them teach classes of 30 regular students, and one teaches 30 gifted students, is B) "catering to a select few"? If so, which select few? Both regular and gifted students receive education adjusted to their abilities, so are they not both being catered to?