No. It's more like you took a burger that someone else owned and prevented him from doing what he wants with it. It is an encroachment on the rights of others.
The thing is I see both sides: I am in favor of private property, and I don't like the idea of e.g. home owners' associations that try to limit what color you can paint your own house. But I also grew up in San Francisco, and I really do not think this city will be more livable with ~100,000 or more people trying to live here.
But then how do we decide who does get to live here? And who gets to build what?
It's not easy, but standing around yelling "You NIMBY!" at your neighbors isn't productive.
My mother lives in a rent-controller home in SF. In effect, her landlord is donating ~$1500 to her every month (comparing what she pays with what a new tenant could expect to pay.) On the same token, does ownership of a building really confer on you the right to make as much money as you can, or are you just the temporary steward of a physical artifact the primary purpose of which is to provide a good life for its residents and the neighborhood? What is the purpose of real estate? What is the purpose of a city?