Most people see it as a non-problem because they assume that they're in a separate property market from the billionaires, and because the billionaires are such a small set of people. What they don't realize is that housing is an extremely inelastic good, so even a 1% supply destruction is going to have major effects on price-- possibly a 20, 50, or even 100 percent bump. It's not about high incomes (NYC incomes are high, but not that high for average people) or some amorphous "people want to live here". It's demand inelasticity coupled with our failure to keep dirty foreign money out. And the ultra-rich and middle-income markets do interact, if not directly. There's a push-down effect. If dirty billionaires are taking apartments that should be "only" $5-10 million at 5x prices, then the working rich get pushed down into upper-middle-class housing, the upper-middle-class gets pushed into middle-class housing, and on down the line, with the working classes getting absolutely fucked and having 1+ hour commutes.
Moreover, in addition to the foreign scumbag money (which is a problem on the West Coast, just as in New York) you have a lot of gray-water foreign money: overseas speculators who aren't criminals themselves but are betting on increased scumbag-fueled demand for U.S. real estate.