> Also, rich people can just buy an old Victorian and modernize the inside, converting a non-luxury housing unit into a luxury one.

Oh, it's worse than that: they can buy an old Victorian full of apartments and convert several non-luxury housing units into a single luxury housing unit. Which is something that tends to be generally encouraged by the zoning and planning codes, because single family houses are regulated much less strictly than any other kind of building, and also tend to be exempt from things like rent control.