I went to Bell High school in Canada as a gifted student, and I can say that I never experienced any kind of bullying - and I was very weird. Everyone was weird. It was great, and very much not like elementary school.
I went to a magnet school and my experience was pretty good. Sure, there were "jocks" - but their "power" is vastly diminished in a school where kids are sad if they don't get A's. As you said, it's like minded people being in the same place.
Unfortunately, not everyone has access to a school like that.