I wonder the same thing as you though, there are so many great cities in this vast country, and we are a highly connected society thanks to technology. Why is physical geography still important? You could set up in another city, and even after plane tickets to Palo Alto to meet with VCs, would still come out ahead.
The other point is exactly what you said. Set up anywhere in the country and fly in to meet VCs. You’d still come out ahead.
If you want to penny pinch and build a long runway, yes, SV is horrible. SV is where you want to go if you want to spend exhorbitant amounts of money to hire top talent (i.e. outbid Google and Facebook) in expectation of making even more money.
Business owners often don't really know why it is working. So once you get something that works, it is quite dangerous to change the formula in any way. You may be killing it and you may have no idea how to remedy it if it goes bad.