It's probably worth it to go there and bank money for a while on the most efficient CoL situation you can manage (and especially to get Stanford on your resume).
With the caveat that the above is definitely the rational thing to do, I will note that I chose not to do it because I couldn't live in that region. Doesn't quite jive with my personality or what I wanted in life or out of my career.
So I accepted a haircut on what I could do monetarily, lived elsewhere, and orchestrated a much more unorthodox career progression. I felt okay with doing that because I'm more or less unconcerned with finding opportunities or generating sufficient wealth in the long run, so maximizing life satisfaction while I'm still young became the priority. From what you wrote, I feel like you're probably in a very similar situation.
You're likely to get a lot of people endorsing SF, so I'll play devil's advocate and list some things that are imo better in NYC and/or LA (Santa Monica) than in SF:
(1) Dating and relationships. The gender ratios are more favorable for men in both LA and NYC, and it's just easier in general in my experience.
(2) Culture and nightlife. Vastly superior in both LA and NYC.
(3) For LA specifically: the weather and proximity to the beach. The other nice thing about LA is that you don't give up on the outdoors advantage of being in California either (Norcal is admittedly great too). NYC is a lot worse for both weather and options for adventuring in nature.
(4) For Santa Monica specifically: extremely relaxed atmosphere and an emphasis on wellbeing and just plain enjoying life. This has influenced me personally for the better. YMMV.
(5) Diversity of people. The Bay Area has definitely become a lot more homogeneous in a way that I find troubling. But by contrast there are some who can't suffer people who think or behave very differently or even irrationally (for example some person enmeshed in Hollywood or in high finance), whereas I feel invigorated by meeting people who are very different from myself, even if I believe they're flawed. YMMV.
(6) Availability of technical talent. It's useful to be in a market of talented people with fewer options in case you ever feel like you want to try to build something on your own.
(7) Cost of living. Slight advantages to be had in NYC or LA, depending on how well you optimize. However, the Bay almost always wins on questions of net income. You can pull some very juicy comp packages in NYC and LA too, and get very close to a wash, if you know how to play the market.
(8) Entrepreneurial opportunity. You're more likely to come across an interesting corner of the world that hasn't been tapped by technologists outside of the Bay Area.
(9) Demand for your capabilities. Tech talent is rarer/sparser outside of the Bay Area and you're consequently a more coveted asset. This goes well beyond career leverage. It's easier to find serendipitous opportunities to meet, help, and even join in something outside the Bay Area partly for this reason.
I'm going to reiterate that the above is my opinion. I'm sure a lot of people would disagree with me on much of what I said.