The one advantage besides talent pool (which is a big advantage!) that both New York and San Francisco have, and that's often overlooked, is proximity to customers. If you're in a business where high touch sales are important, or where you need to be able to meet regularly with your customers, it's very difficult to do remotely.
I'm not sure if the cost-benefit really plays out in New York's favor. I often think that for what it costs to live here, it might be cheaper just to fly in twice a month from someplace else. But it's certainly an advantage I underestimated when I started out.
Downtown is not quite New Jersey, but Midtown certainly seems to be the hotness these days.
At the same time, the older generation, such as my parents, used "New York" (or "the city") to mean approximately "Manhattan", a holdover from what the city was before the consolidation with what are now the remaining boroughs. That always seemed old-fashioned to me. I believe you can still see some old subway signs that use "New York" to mean "Manhattan". (Incidentally New York is also the name of the county that is almost but not quite coterminous with the borough of Manhattan.)
When I was growing up, the state was more or less never referred to as simply "New York" by people in the New York the city. It was "New York State", most of which was "Upstate New York" (though that couldn't refer to Long Island, and it was never quite clear where 'Upstate' began).
The reverse shibboleth quality of "New York City"/"NYC" is probably dying out, may be already dead. Nonetheless I find it annoying.
Wikipedia even notes the differentiation is important.
"The city is referred to as New York City or the City of New York to distinguish it from the State of New York, of which it is a part"
And I've lived here almost a year now and have never had a cockroach in my apartment, or even so much as a silverfish in the bath tub.
I'm sure you can find a really tiny non legal space in Chinatown or similar but I'm not sure I would call that a bargain.
The data proves that the kinds of companies that need a great engineering talent pool and venture capital will tend to hail just a couple of locations.
I'm not saying there aren't exceptions. There are. But in an endeavor where the failure risk is so high in the most optimal of circumstances, I wouldn't add another risk factor like location into the mix.
Just my thoughts.
Unlike NYC, in Upstate NY rent and required salaries are lower - it's a great place to start a company (though I may be a bit biased).