Sure you need to be able to talk to other groups of people but not to the extent that you require 35 hours a week of education along side them for years. Additionally I'm not sure being educated alongside them actually helps. There are a a hell of a lot of intelligent people who become resentful of that situation and that attitude can take years to mature out of, if they ever do.
I ended up teaching my AP Statistics course. It was ludicrous: as a junior I somehow understood the material better than the teacher and was expected to teach a class full of seniors.
Yes, it was more educational than the alternative (sitting in a corner reading a book while totally phoning it in) but I also have no doubt that I would've learned far more in a course which actually challenged me.
---
Also, this notion that our real world experiences are anywhere near as heterogeneous as public schools is downright ludicrous. It's not like we hire random people off the street to be software engineers. The average IQ of everyone I interact with is well the national average.
Honestly I think the worst part is when teachers would get annoyed that you weren't 'paying attention' and force you to stop doing the crap that's keeping your mind busy and 'focus on school', even though you already get pretty much straight A's.