I think you're free to use your money to buy yourself out of discomfort and into a nice place for your family, but to gate-keep others out of the freedom you enjoy is wrong.

Yes, you might have to face discomfort so that others can afford to put a roof over their head. There is a housing affordability crisis in North America for which it is necessary to build (on top of many other efforts required).

Yes, it's completely rational for you to be a selfish actor in this instance. But I don't believe it's rational for government to continue to support NIMBY'ism for the long term health of its society.