Your example is true in as much is it discredits your overall stance. A, if not the, major cause of housing crises (meaning keeping people affordably sheltered, not as in housing bubbles) is zoning law.
At its best, it keeps industrial sites from being built within certain residential areas. At its meanest, it's used to create economically and racially segregated communities, oust families from their homes, and freeze cities in time. Upholding its use for any of the latter could generally be assumed to be the result of particularly stubborn and selfish mindsets towards one's own convenience.
Also troubling about current US zoning practices are its weird exclusive use-per-zone rules, the threat to one's right to unencumbered use of their land, arbitrary development cost increases within cities (contributing to sprawl), and rent increases from zoning exploitation to artificially raise a community's median property values, and, as a side effect, driving up the rent in surrounding areas.
I consider these and any act which would potentially lead to the eviction of someone from their home to be much larger threats to the availability of equitable and affordable housing than people with the wealth to be mobile being bothered by not having the right to control what others do on and with their own property.