Have you ever been inside a school? The stupid kids don't gather around to bask in the glow of the gifted kids. They DO gather around them and call them 'faggot' and trip them in the hallway, especially if they are smart but not athletic.
As someone who has been bullied in the past, I never thought the solution was avoidance. Our schools have to improve on all counts. Segregating students on any criteria is the opposite of the solution.
In a perfect world we would make all our schools great, but I think you'll find that this is harder to actually do than it is to say, and in the meantime it is very easy to get great teachers to work with gifted students.
Imagine you are a great principle/counselor/teacher. You can work anywhere, because you are great at your job. Do you choose to go to the school with the superintendent that is nearly illiterate and completely incompetent at the district where the kids regularly assault each other and the staff? Or do you choose to go to the school where the kids love to learn, love to read, love to study, and respect their teachers and their school?
If for no other reason, you don't want the cognitive elite going through life never knowing that the rest of the country are reasonable, functional people instead of fools and rubes.