I think this is why we break into groups like marathon runners, 7-minute milers, car-to-car mountain climbers, etc. In the forming of this group is also an "implicit" social contract about what it takes to belong. The existence of these groups ensures that the lowest common denominator meets a certain criteria in order to join.
Edit: Additionally, the recent moral philosophy is that excluding people from a group is unfair. College, the C-level suites, Everest, marathon running should be accessible to everyone. I think the backlash now is that previously closed groups feel like they are forced to let people in below their standards and thus forced to sink to the lowest common denominator in society.