That is what happens when only one landowner chooses "defect". If all of them do it, then 20 people times the number of lots might find a place to live, except it's no longer the kind of place where they want to live.

Or put another way, there is already a lot of places where you can live cheaply, it's just that a lot of people would rather live in Palo Alto than in those places. But if you turn Palo Alto into a housing project, then the reason why they want to live there might stop existing, because all the rich people, who are the reason the rest of us want to be in the Silicon Valley, likely prefer not to live in close vicinity to "bigger buildings filled with tinier apartments."

Of course I don't have a glass ball and cannot know for a fact that's what would happen, but it doesn't seem far fetched and at least personally I find it no less believable than many other tragedy of the commons situations which are posited to justify this regulation or that.