I think that HN, while not a college, might be a counterexample for this idea.

Here everyone can join and communicate, and yet we can maintain a high level of communication. The ones who don't fit are quickly flagged or simply won't come back again.

I think selectivity is good, however it doesn't need to be enforced in the admission step, it will happen in the middle of the process. People who would not be able to follow the course requirements would quickly drop out.

I'm not arguing that this idea would work (I don't really know), but based on the HN experience I think it should be considered.