One thing to notice is that plenty of industries have followed this pattern until they became mature (cars, breakfast cereals, aircraft to some degree). And you see this in software: the pressure discussed is at the frothy end of the market.
> it’s a just so story commingling
Explanations:
In science and philosophy, a just-so story is an unverifiable narrative explanation for a cultural practice, a biological trait, or behavior of humans or other animals. The pejorative nature of the expression is an implicit criticism that reminds the hearer of the essentially fictional and unprovable nature of such an explanation.
com·min·gle: mix; blend
Also it's not worth it where the valley is behind the curve (e.g. engines are designed in different centers of excellence).
Even for venture-backed (intended-to-be) high-growth companies, the Bay Area is only about 50% of all US venture capital investment. And there has been some spread: in my professional lifetime the "Valley" has grown to encompass SF and parts of the East Bay. When I first started working here people lived in SF and commuted to tech jobs in MP and points south.
But some products are just so high margin (iPhones, Internet search, USB-powered vape tubes) that it's cheaper overall to be here and take advantage of the affordances available.
Aka factories minimize distance between inputs, other factories, and customers. So Coca Cola builds multiple factories because buying water locally and shipping the finished bottle 10 miles is cheaper than shipping it 1000 miles. For car factories there are some costs for traveling in between factories so most of them are still clustered in certain states, but the factories can be spread out. Plus, once a city has too many factories increasing wages, the next company that comes along will build their new factory elsewhere.
Tech offices have effectively zero distance to customer cost (at least within the USA, you might build another place for say the eu) So the main costs to minimize are input costs (employees) and costs between offices (coordination).
Unfortunately the "cost" between offices is relatively high. Unlike multiple factories, aka ship the plastic bottle from one factory and the plastic cap from another factor to the main bottler factory, multiple offices require coordination with one another. Or to put it in tech words, it's not easily parallelizable. The more offices spread out in multiple cities you have, the more meetings you need to coordinate between each city.
And the cost between offices is essentially "time" the most precious resource for a tech firm, which cannot really be removed. And so tech firms pay the higher col rather than spreading out into other cities; well until the col really increases too much.
Remote work is a viable option. Some "remote first" companies have been very successful. And ... there are lots of talented people who do not live in SF.
Perhaps it is the founders and VCs that are clinging to the status-quo.