In doing this, you are no longer an owner, thus no longer have a vested interest in limiting housing.

You profit from this, but at the expense of your neighbors. Imagine the opposite scenario where all over your neighbors do this, but you're the single hold out. Would your home be worth more, less, or the same with twice (or more) the number of people living on the block? Obviously, we don't really know, but the holdout is likely fearful that their property value would go down dramatically.

You can see why people who have $1,000,000 mortgages don't want to see the value of their house fall to reasonable levels, because they'd be the ones taking the six-figure bath.