Imagine what would happen if this didn't happen... imagine living in a place as inexpensive as a medium-sized Midwestern city, but with Silicon Valley salaries. Imagine how many other places that wealth could go: more startups, more gadgets, better infrastructure, world-class everything, better health care, even generosity. But nope, real estate just soaks up wealth beyond a certain point and redirects it back into the financial pyramid.That sounds at least a little bit like where I live now. We don't have SV-level salaries, but the tech industry can pay roughly $60k-$80k per year in a place where $45k is the median dual-income household and $33k is fairly comfortable for a single adult.
A few key differences from, say, most of the United States, that I think can transfer over to other places without getting into local culture:
* There's no state/provincial/regional government, except in rural areas, where a group of villages is governed by a regional council. If you live in a city, you've got the municipality and then the national government. One less layer of abstraction, and the municipality is entitled to take decisions that in the USA would fall to the state-level government.
* The municipality levies "arnona" (our word for the old English "rates" system) as a tax on the square-meterage of dwelling units. The occupant pays the tax, but like a VAT, the arnona is factored in as part of the price you pay for a rental unit. The effect is to discourage the development of very large dwellings in favor of dense buildings full of smaller dwellings. The other effect is that we don't have conflicts over assessments of property values, because they're not taxed. Living in a beautiful, well-located house won't stick you with extra property taxes unless the city specifically decides to zone your area for a higher per-square-meter rate. It works pretty decently, overall, although there is a problem of schools being underfunded (though this may be a political/national problem rather than structural).
* Urban planning here is pro-urban rather than anti-urban like in the States. The typical family dwelling is a three-bedroom apartment (one master bedroom for the parents, one bedroom each for your two children), building is planned in terms of semi-self-contained neighborhoods, and zoning consistently sprinkles in a certain amount of parks, a certain amount of schools and clinics, a certain amount of community centers and houses of worship, a certain amount of sports fields, and a certain amount of mixed-usage residential/commercial/office space per neighborhood.
* Public transit here is not just for poor people. Bus and minibus systems operate extensively, including between cities, and there's also a coastal intercity rail line and a small subway that goes up and down the mountain we're built atop. I estimate I can reach anywhere in this city that I might work within 30 minutes of automotive commuting, 30-60 minutes on the bus system, and that's if I can't actually walk to it.
* NIMBY is not a thing here. Period. Only major real-estate developers go around actually thinking their land holdings are so freaking important that they should have the power to block other people's projects, particularly municipally-sponsored ones. There are whole neighborhoods built around the shipping port, the oil refinery, and the High-Tech Park where the BigCos live.
* As far as I know, you can't actually build things outside cities or towns. The countryside is countryside, and nobody is necessarily entitled to land rent or real-estate tax from it. So big things like tech parks and office buildings get built in cities, because you have to build them inside some municipality recognized by the national government and once you have that requirement, you might as well put it near your prospective employees and customers.
TL;DR: No, cities and urban living do not innately cause ultra-high real-estate prices. Yes, public policy matters, because urban planning, transit infrastructure, land taxation, and a strength of property titles appropriate to urban life all provide rules and incentives for building a city that's actually nice to live in. There is absolutely no reason a place like Silicon Valley can't make itself a very nice and even affordable place to live, except that the existing residents enjoy feeling privileged. They get to feel privileged, because in SV and in many other desirable American places (Boston-Cambridge and New York City come to mind), real-estate has become a status good with a permanently fixed supply. They'd feel somehow insulted to find out that their land is a mere commodity, to be bought and sold in bulk as common families want or need.