I'll second the points about yale culture. Honestly, if you're worried about being A downside would be continuing to be the “too-intense guy” I come off as at my high school, and potentially continuing to feel slowed down by my peers." then if you're as technically / mathematically oriented as you claim, outside of certain parts of mit, caltech and other places with a large engineering culture, you'll continue to be viewed as such.

A simple, but in retrospect insightful bit of advice that I ignored (but shouldn't have) when choosing a university is the following: go through the course catalogue and figure out what percentage of all the classes you would certainly find interesting, all other considerations being equal, choose the school with the best percentage!