Not sure about what anyone here considers as libertarian, but if you were to ask a libertarian what they would tell you is that if you own property you are free to do as you please so long as you do not harm someone else.
Also not sure what you mean by lefty orgs, because labor unions, public unions and environmental groups are the biggest lobbyist influence on the high price of housing. When it comes to affordability, housing is generally (ahem - not "always") less affordable in blue states and least affordable in the bluest regions of the blue states.