Vancouver has built tower after tower after tower, it's not recognizable from 25 years go. Also - it's not the responsibility of local residents to demolish their homes and move into flats 'so you can live there'. The NIMBYism argument is rubbish as you have no right to influence or dictate the terms by which some other community wants to live.
If people want to live in homes instead of jammed into skyscrapers Hong Kong pr NY style - which many would view as a horribly reduced quality of living - it's their choice.
NY, HK - two of the most densely urban place in he world have not solved the 'housing crisis' - they're among the most expensive places in the world to live.
That said, cost of living has skyrocketed far beyond what can be attributed to that - fairly high migration, a lot of foreign buyers (only 4% directly, but 'kids with citizenship using parents money' amounts to a big other cohort) and that is very easily enough to drive prices. Combined with low interest rates, an aging population with incredible voting power/influence, a booming economy and there you have it.
'Globalization' means SF and Van. basically are 'up for grabs' by everyone on the planet, not just locals.
1/2 of the real estate in the most posh parts of London are owned by foreigners for he very same reason.
There needs to be a total rethink on how taxation works in those parameters.
That said, it's rational: if a lot of talent wants in, prices will be higher. I don't know if 'artists' are going to be able to live there without some kind of very specific interventions.
Other things, such as public transport can have a huge effect - if SF Bay were to be amalgamated into at least some kind of 'mega city' you might see high speed commuter trains.
Imagine a very fast train from Stockton, Pleasanton, Gilroy, Santa Rosa - getting you to 'wherever' in the Bay area within 25 minutes?
There's tons of real estate around the Bay Area, for artists, working class, middle class, regular folks, people who want 'big houses' etc.. it just not 'accessible' to the core.
Or even micro/satellite offices as a civic mandate: building out offices that can be adjusted in size quickly, so that Salesforce, FB, Snap etc. will all have offices in San Jose, SF, Sacramento and bounce back and forth as needed.