The problem is the lack of housing supply. Get rid of the ridiculous regulations, allow more stock to be built, and revel in all the free foreign money that's being showered on your city.
Yes, more housing stock is going to change the "charm" of your little haven. Guess what? Cities change. All of your precious cities are nothing like they were 50 years ago. That's how life goes. Every alternative you have results in a change to the status quo: (1) do nothing and watch all the artists and misfits (then the middle class) get priced out of the city, (2) impose even more regulations to try to keep rents affordable (which is never a solution, see San Francisco), or (3) build more housing.
Only one of these solutions makes any sense. But, no surprise, it's always easier to blame the foreigners than it is to risk your political capital.