The only unique thing about SV is the unenforceability of non-competes, that was clearly absolutely critical in the original silicon days (https://en.wikipedia.org/wiki/Traitorous_eight) and probably sped things up by decades, I've also experienced two good pieced of technology killed by them outside of California, and I'm frankly left without a better explanation.
Depending on who wins out this general election, we will possibly see a rather ironic shift that will not only undermine the hegemony of Silicon Valley, but will also make staffing startups with low cost talent from all over the globe far more feasible. I've been telling Silicon Valley for a while now, be careful what you wish for, because it's easy to lose sight of the real world from within a rose colored bubble.
I have a feeling that as/if globalism were to expand and consolidate power and control, it also broadens, dilutes, and devalues "labor" on a global scale as you start feeling the effects of globalism that clothing makers, furniture makers, auto makers, etc. have all felt before you. The only thing that is keeping the tech industry enslaved to Silicon Valley, is the dependence on their VC masters, as you so well described. Once that is no longer a benefit or necessary or an advantage, what keeps people in Silicon Valley?