If your goal is primarily to save money so that you can retire or don't have to work or can choose when/where/how to work, you're already on the right path. Economically, demand for housing in the bay area is strong enough right now that rents are rising to whatever level people can afford to pay. That means you're basically assured of having little or no surplus unless one of your ventures is wildly successful.
If your goal is to develop a bigger network of people in your industry, you're probably missing out, although your client network will be very valuable to you as well. Also, not all of the people you're missing out on are people you want in your network anyway...
The culture in "silicon valley" is toxic. People are obnoxious and narcissistic; many are pathological liars and self-promoters. Plenty has been written about this, both serious and satirical. There are certainly people who enjoy it, but it sounds like you're fairly happy where you are, so you probably aren't one of them.
If you're tired of bootstrapping and want to get funded, you'll probably have a better chance in the bay area or some other large city. Otherwise this is moot for you.
Also, remember that most of "silicon valley" isn't involved in technology any more. It's become mostly a media/advertising center. There are still technology companies around, but the focus has definitely shifted.
You might want to ask yourself not "am I missing out?" but rather "what would I like to change about my present circumstances?" and go from there.