Schools are not communities, they have communities within them, but they are more than just communities.
Your high school homecoming court also has a shared identity and a set of experiences that people outside the court don’t have. How is either important?
A lot of people paid $75 to become one of those data points, and the majority of them didn’t pick Caltech because of the social opportunities (because Caltech is not anywhere near the top of that list unless you’re talking social opportunities strictly for people north of 180 I.Q.).
In Caltech we have an exclusive and highly desirable learning institution with a finite and insufficient number of seats. Rigor or optics, you can’t have both. If you’re choosing optics, own it and don’t be ashamed of it.
It's not like they're taking women that can't hack the coursework. They could replace the entire incoming class with select people amongst the rejected and the class would still be successful. College admissions is partly a crap shoot. If they tilt the crap shoot part in a way that makes the community better, who cares?
Everybody's all for tilting the crap shoot until they start tilting away from Asians.
Relatedly, a selective school means that the peers are likely to be similar. Those connections can last a life time, and socialization can greatly broaden your knowledge base and lead to a potentially more interesting skill set and/or a more complex trade.