Agreed. Also, rich people can just buy an old Victorian and modernize the inside, converting a non-luxury housing unit into a luxury one. That's one less unit on the market for normal people to buy or rent.
For example: http://sf.curbed.com/2015/10/8/9913302/victorian-disembowele...
This is what happens when you don't allow development to proceed. It only makes things worse. Rich people will get their luxury housing, one way or another. The rest of us are left with fewer options.