I couldn't agree with you more. I actually said this at my last job interview when asked why I was leaving. I've learned more in the last 6 months being surrounded by much smarter people than the 6 years of being "the smart one".
I won't lie though, the ego-hit of being "the dumb one" when surrounded by truly brilliant people is hard - 100% worth it - but hard.