OP lives in Austin. If you have $1M net worth, you can probably buy a house there and live there. If you don't go do that, I think you're just punishing yourself.
Meanwhile, if you live in the Bay Area, I can totally see "$1M net worth but not really able to buy a house".
Where you choose to live in really does matter a lot. Not to say there aren't issues with property costs, but... come on.