Disclaimer: I live in Chicago, but started my company in San Francisco. I've been in gravitational orbit around the SF Bay Area my entire working life. (For a majority of the past 6 years, I've spent 1 to 2 weeks a month in SF.)
The standard arguments for starting a company in California is:
1) You'll be closer to venture capital.
2) You'll be closer to talent.
3) You'll be closer to customers.
1) If you want or need venture capital, then it helps to play by their rules. One rule is: "Locate your company's headquarters within an hour's drive of my office." In my opinion, this is stupid and short-sighted, but from their perspective, it kinda makes sense. Giving you capital means (most likely) also sitting on your board of directors. And if those board meetings only require a short drive (instead of flying), then their life is easier. Also, if they're giving you a ton of money (many millions, perhaps), they'll sleep better at night knowing you're close by and can meet in person at any time.
2) Access to talent is closely related to access to capital. VCs will want to give you money only if you want to get big and make billions of dollars. (AKA: Are you the next Google, Facebook, or Apple?) The implication is that if you're going to scale big, you're going to need a ton of employees. If you're based in the middle of nowhere, your growth potential is limited to the size of your local talent pool. Again, I think this argument is silly and short-sited. But I've heard VCs make this argument directly to me in the past.
3) It generally makes sense to be close to your customers, especially early on when you don't have any. If you're a software company selling to other software companies, it might make sense to be based in San Francisco, regardless of how expensive it is to operate there. (You're going to make billions, anyway, right?) If your early customer is the U.S. Government, it might make sense to be primarily based near D.C. If your customer is the media, it might make sense to be based in New York or Los Angeles. If you can argue that there isn't a central cluster city of potential customers for your business (or you can argue that it doesn't matter), then it's less likely you'll need to move to be closer to them.
This is how I plan to do my next startup given the above three arguments:
1) Don't plan on needing VC. Bootstrap and get to profitability as quickly as possible. Then I won't have to play by VC rules and move closer to them.
2) Don't plan on needing to grow big. Plan on hiring people wherever they are, and create a remote-friendly working environment.
3) Live near an airport. As long as I'm close to an airport with lots of direct flights, I can visit potential customers whenever/wherever I want. (This is why I'd rather be in Chicago near O'Hare Airport, than live in the middle of Kansas, for example.)
However, I still might move back to the Bay Area someday. But I'd only do so for personal reasons - e.g. being closer to family and access to awesome hiking trails.
[edit: spelling and grammar]