Not true - there's plenty of places where most of the people you'll meet are quite intelligent, and not everyone in college are college-level smart. If you go to Harvard, sure, it's all fairly smart people. The average state school? Not so much.
As for places with lots of smart people, it's very worth it to go get coffee at the Ritz Carlton sometimes. It's $10 for a pot of coffee that'll serve 2-3 people, so it's only like $5 to go if you go with a friend or the girlfriend, and it's good coffee. There'll be smart people there from out of town, often having a quick bite to eat alone while reading a newspaper. Contrary to popular belief, wealthy people tend to be very friendly and approachable if you're polite, complimentary, and interested in striking up a chat - I've met a fair few intelligent people while having coffee at a nice hotel or ex-pat bar or other place like that.
If you're willing to try to charm a little bit too, you can probably talk your way into the business class/first class lounges with decent success when you're flying, and people inside tend to be a smart, a bit bored, and quite receptive to having a conversation.
There's lots of places to meet smart people. University is good for some smart people, but not all. I enjoyed some of the business and technical classes I took, but absolutely went nuts with the "general requirements" and the rote busywork it entailed.
So there's lots of places to meet smart people besides university. And seriously, I'd recommend anyone who isn't comfortable in "wealthy places" to go have a pot of coffee at the Ritz Carlton. It's $10, cheaper than a mid-end restaurant lunch. Strike up a conversation with someone. Might change your life, but at the very least you get more comfortable in the environment.