I was going to guess Caltech which has a very similar policy and culture as described. The only thing that didn't fit was the "women's college across the street".
Yep. Also I am compelled by ancient tradition to point out that Caltech's style and traditions are going down the tubes due to the shift from administration by faculty to administration by professional administrators. (Note that this has been the received wisdom for long enough that it can't possibly be quite as true as it's made out to be.)