Dropping the price of a house by a few percentage points can be the make-or-break for some families. And small changes in availability can have large impacts on price.
If we banned (or severely penalized) all entities from owning more than 5 residential homes, this would probably reduce cost by a few percentage points across the board. That's thousands of dollars.
Personally, I think unoccupied homes in general ought to be penalized (beyond just tax burden, an actual vacancy tax).